{'fraction_non_alphanumeric': 0.02005027181855381, 'fraction_numerical': 0.004150347810837669, 'mean_word_length': 5.1275071633237825, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Creating learning health systems and the emerging role of biomedical informatics

Abstract Introduction The nature of information used in medicine has changed. In the past, we were limited to routine clinical data and published clinical trials. Today, we deal with massive, multiple data streams and easy access to new tests, ideas, and capabilities to process them. Whereas in the past getting information for decision‐making was a challenge, now, it is how to analyze, evaluate and prioritize all that is readily available through the multitude of data‐collecting devices. Clinicians must become adept with the tools needed to deal with the era of big data, requiring a major change in how we learn to make decisions. Major change is often met with resistance and questions about value. A Learning Health System is an enabler to encourage the development of such tools and demonstrate value in improved decision‐making. Methods We describe how we are developing a Biomedical Informatics program to help our medical institution's evolution as an academic Learning Health System, including strategy, training for house staff and examples of the role of informatics from operations to research. Results We described an array of learning health system implementations and educational programs to improve healthcare and prepare a cadre of physicians with basic information technology skills. The programs have been well accepted with, for example, increasing interest and enrollment in the educational programs. Conclusions We are now in an era when large volumes of a wide variety of data are readily available. The challenge is not so much in the acquisition of data, but in assessing the quality, relevance and value of the data. The data we can get may not be the data we need. In the past, sources of data were limited, and trial results published in journals were the major source of evidence for decision making. The advent of powerful analytics systems has changed the concept of evidence. Clinicians will have to develop the skills necessary to work in the era of big data. It is not reasonable to expect that all clinicians will also be data scientists. However, understanding the role of AI and predictive analytics, and how to apply them, will become progressively more important. Programs such as the one being implemented at Wake Forest fill that need.


| INTRODUCTION
The nature of data and evidence in healthcare is experiencing rapid change. Personalized healthcare, making decisions that are more likely to benefit the individual, is evolving and requires more kinds of data than has been used until recently. Personalized decision making requires moving beyond the traditional data sources to understand the individual more fully. 1 The techniques used in the past, such as the null hypothesis analysis in randomized controlled studies (RCTs) are still important, but insufficient to achieve personalization. The myriad streams of data that may influence health (environmental, domestic, ethnic, cultural, access, political views, wearables, genomics, etc.) leave us in the position of not understanding which of the streams is more important or how they interact. The recent creation of the sub-specialty of clinical informatics confirms that more sophisticated approaches, such as artificial intelligence and machine learning, are required to use both the usual clinical data and real-world data.
Clinicians must become adept with the tools needed to deal with the era of big data, requiring a major change in how we learn to make decisions. 2 Innovation in health information technology, such as the electronic health record (EHR) has often been met with resistance and frustration because it required extensive training, did not support clinical workflow, required additional time or interfered with interaction with the patient and did not provide sufficient value.

| REQUIREMENTS OF A LEARNING HEALTH SYSTEM
In the past, the evidence for decision-making developed slowly and episodically. A topic was studied with a series of RCTs with the result becoming the standard until the next study was published. The feedback loop to evaluate the impact could be months or years long. Such a lengthy process does not support personalized healthcare. With sophisticated analytics and rapid access to data we can update the system with ongoing feedback. A "Learning Health System" embodies such a loop. Data is converted to knowledge, knowledge is used to perform an intervention, the result of the intervention provides new data, and the process cycles continuously.
The Learning Health System is elusive is often described by its characteristics 4 : 1. Every patient's characteristics and experiences are securely available as data to learn from.
2. Practice knowledge derived from these data is immediately available to support health-related decisions by individual members of society, care providers, and managers and planners of health services.
3. Improvement is continuous through ongoing study addressing multiple health improvement and related goals. 4. A socio-technical infrastructure enables this to happen routinely, with a significant level of automation, and with economy of scale. 5. Stakeholders within the system view the above activities as part of their culture.
Working effectively in a Learning Health System requires education and training. We are dealing with more data, and more kinds of data than ever before. Identifying the valuable data and discerning patterns that allow better decisions is a new skill. We are moving in that direction, and Biomedical Informatics, particularly clinical informatics, is key in those efforts. Clinical Informatics is the newest clinical sub-specialty, and training programs and fellowships in informatics are becoming common. Creating the environment to support the development of a Learning Health System is an important goal in which the WFBMI is an active participant.
A by-product of the widespread digitization of healthcare over the last two decades is the large amounts of "digital exhaust" (ie, information collected from the internet by a person or an organization) from technology supporting the clinical and administrative process of delivering care. In fact, as in other systems outside of healthcare, the creation and existence of data is exponentially growing. 5 This data can be leveraged to the benefit of patients, families, providers and staff, as well as the healthcare system at large. One of the main challenges that now exists is how to process and analyze the data to gather actionable insight and knowledge from it. One can easily get lost in "big data" when mining it, hence the need for data scientists and domain experts well-versed in healthcare processes; clinical informaticians. Without being familiar with the clinical delivery system, the data will not be viewed in the right context to allow this synthesis to occur, and for intervention opportunities to become clear.

| Examples of using data to create knowledge to improve practice and operations
Horwitz et al 6 have published a prime example of how to systematically identify opportunities to apply LHS principles and create value from those activities. In their approach, they largely applied quality improvement and data-driven approaches to achieve favorable outcomes without expending a large number of resources. Much of their value came from "exnovating" or removing unproven activities that required resource spend for what was discovered to be no gain. We are following their lead, and beginning our own process, as are many others.
Electronic Health Records (EHRs) are tremendous sources of data, and clinical decision support (CDS) in particular is a rich target for much of our work. In 2019 alone, there were 29 million best practice advisories that were visible to providers and staff, many of which were not acted upon and only served to generate noise in our clinical care delivery processes if they were providing no value. We have begun to systematically and rigorously examine the data for the best In the example above, data that are readily available is being used to generate insight about practice patterns, which is then used to create practice efficiencies and optimize provider time. By utilizing data to identify instances where care is not being optimized, but is creating useless, inefficient "busy work," we are replacing that work with the freedom to do other activities to improve patient care, while improving provider satisfaction and removing a source of ire of physicians. 7

| DEVELOPING A BIOMEDICAL INFORMATICS PROGRAM TO HELP AN INSTITUTION'S EVOLUTION AS A LEARNING HEALTH SYSTEM
When we created the Biomedical Informatics program at Wake Forest School of Medicine (WFSoM) in 2018, we designed its structure to help Wake Forest's evolution as a Learning Health System. Wake Forest's vision is to be "a preeminent learning health system that promotes better health for all through collaboration, excellence and innovation." This cross-disciplinary initiative is designed to integrate • Educate: Support and inspire the training of the next generation of investigators in the principles and practice of biomedical informatics WFBMI's plans for innovation through artificial intelligence 8 will take many forms. For example, building and implementing EHR based clinical alerts, decision support tools, structured data capture, order sets and facilitating pragmatic clinical trials by being able to randomize by clinics/hospitals. Some driving projects include computer-assisted assessment of otoscopy imaging 9-11 grading of follicular lymphoma slides, 12-14 chest pain risk stratification, 15 prediction of glycated hemoglobin values, 16 and an EHR based frailty score. 17  While patients have the opportunity to discuss the options with the clinician during the visit, ultimately the patient "self-orders" the preferred screening test (eg, fecal blood test or colonoscopy). Patients who self-order a colorectal cancer screening test receive text messages to remind them about the test, to provide encouragement, and to ask if patients have questions. Subsequent text messages are sent to patients in order to encourage completion of the screening test. A randomized controlled trial demonstrated that screening was twice as likely to be completed (30%) in the intervention group as compared to the control group (15%). The design of this project was built on research showing that multi-level CDS tools are more likely to be effective. 20 For example, Roshanov showed that CDS systems were much more likely to succeed when the tools provided advice for patients in addition to practitioners. 21 The study design should be given significant credit for the success of this project and highlights the need for informatics professionals who are familiar with the litera- Research Networks (CDRNs) by conforming local data to these networks' Common Data Models (CDMs). We are in the planning phase of utilizing Epic supported FHIR resources as an additional mechanism for data transfer to the TDW. WFBMI is also committed to developing a semantic framework that will minimize data misinterpretation and discovery challenges. We maintain a terminology server to house Uni- In addition to staff-facilitated data requests, investigators can perform queries on the TDW using i2b2 and, upon appropriate IRB approval, they can download automatically generated datasets through a self-service tool. This internally developed tool interfaces directly with the IRB system to determine authorizations and can deliver Excel exports of data in minutes, greatly reducing turnaround times on data retrievals. In addition to EHR data, the TDW has been linked to outside data sources, most notably US census data and the NC State Death Registry. TDW is also our conduit for participation in the federated data networks for obtaining larger sample sizes (particularly important for rarer disease investigations) and realization of the full potential of population research. WFBMI enables WFSoM's involvement and contributes to regional and national CDRNs

| EDUCATION AND TRAINING IN A LEARNING HEALTH SYSTEM
The LHS is a new paradigm for the academic medical center requiring a more diverse skill set for faculty to be successful in clinical care, The 2019 class worked together to extract and analyze data to create a model for predicting pneumonia readmission risk that was presented at the Society for Medical Decision Making. 22 The 2020 class featured 26 student enrollees and involved hands-on machine learning with R to develop risk prediction models from data extracted from the EHR. The course was shifted from an in-person to a virtual format via WebEx due to the Coronavirus pandemic, but the majority of students who completed the course still rated it well overall and would recommend the course to a colleague. One student even expressed interest in re-taking the course in the future. Some of the students reflected that they had insufficient R programming skills to receive maximum benefit from the course. The course is synchronized to occur immediately following a preexisting R beginner course for those students that need the additional training. The WFBMI faculty have also developed a Master of Science level course entitled, "Introduction to Biomedical Informatics" that has been offered to our Translational and Health System Science graduate students beginning in 2019. The overview course focuses heavily on health informatics and includes topics on medical decision making, clinical decision support, cloud-based computing, image analyses, and common data models. The 2020 course received overwhelming interest and needed to be capped at~30 students.  EHR vendor-based certification -The CTSI will begin providing funding for physician-researchers to become certified through our EHR vendor to build optimization tools. This training will make these investigators more aware of the EHR capabilities to support both clinical care and research. It is hoped that this knowledge will facilitate LHS research that sits on the boundary of Quality Improvement and Clinical Care. We are working with IT in order to integrate the trainees with existing projects to provide hands-on experience after certification. IT will benefit from the clinical knowledge being provided to this collaboration.

| The Wake Forest clinical scholars in informatics (CSI program)
Learning health systems require a constant feedback loop with input from a learning community. It is essential to train physicians to be competent in the fundamental components of biomedical informatics, data science, and applied clinical informatics. In an academic learning Analyst to ensure all components of the applied informatics intervention including EHR build/configuration and EHR data extraction and visualization can be accomplished. A comprehensive list of peerreviewed posters, presentations, manuscripts and grants can be found on the CSI website. 23 The CSI is a low-cost, novel applied clinical informatics resident pathway, which can be broadly applied to most internal medicine programs across the United States. There are clearly bidirectional net benefits for the resident physicians and for the health system as a whole. The residents benefit through the experience of driving an applied informatics project end-to-end and deriving academic capital.
The health system benefits by implementation of a locally relevant applied clinical informatics project with robust evaluation. Additionally, CSI functions as internal development and recruitment for broadbased informatics expertise across the enterprise. Future directions include scaling to additional departments, formalizing the curriculum with Wake Forest graduate medical education and defining a sustainable funding stream (institutional vs extramural). 23 The Wake Forest School of Medicine WFBMI is engaged in a multi-faceted program to improve clinical-decision making and health care operations by preparing for the data-driven future. The program emphasizes data acquisition, management and analysis, as well as training clinicians to thrive in the big data era. Value-based healthcare is dependent on such programs.
We are now in an era when large volumes of a wide variety of data are readily available. The challenge is not so much in the acquisition of data, but in assessing the quality, relevance and value of the data. The data we can get may not be the data we need. In the past, sources of data were limited, and trial results published in journals were the major source of evidence for decision-making. The advent of powerful analytics systems has changed the concept of evidence. Clinicians will have to develop the skills necessary to work in the era of big data. It is not reasonable to expect that all clinicians will also be data scientists. However, understanding the role of AI and predictive analytics, and how to apply them, will become progressively more important. Programs such as the one being implemented at Wake Forest fill that need.