{'fraction_non_alphanumeric': 0.024343821228000093, 'fraction_numerical': 0.0062974363794440396, 'mean_word_length': 5.344364937388193, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enhancing the use of EHR systems for pragmatic embedded research: lessons from the NIH Health Care Systems Research Collaboratory

Abstract Objective We identified challenges and solutions to using electronic health record (EHR) systems for the design and conduct of pragmatic research. Materials and Methods Since 2012, the Health Care Systems Research Collaboratory has served as the resource coordinating center for 21 pragmatic clinical trial demonstration projects. The EHR Core working group invited these demonstration projects to complete a written semistructured survey and used an inductive approach to review responses and identify EHR-related challenges and suggested EHR enhancements. Results We received survey responses from 20 projects and identified 21 challenges that fell into 6 broad themes: (1) inadequate collection of patient-reported outcome data, (2) lack of structured data collection, (3) data standardization, (4) resources to support customization of EHRs, (5) difficulties aggregating data across sites, and (6) accessing EHR data. Discussion Based on these findings, we formulated 6 prerequisites for PCTs that would enable the conduct of pragmatic research: (1) integrate the collection of patient-centered data into EHR systems, (2) facilitate structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows, (3) support the creation of high-quality research data by using standards, (4) ensure adequate IT staff to support embedded research, (5) create aggregate, multidata type resources for multisite trials, and (6) create re-usable and automated queries. Conclusion We are hopeful our collection of specific EHR challenges and research needs will drive health system leaders, policymakers, and EHR designers to support these suggestions to improve our national capacity for generating real-world evidence.


INTRODUCTION
Pragmatic clinical trials (PCTs) are randomized controlled trials designed for generalizability, often involving multiple clinical sites with broad eligibility criteria. 1,2 Their advantage is the ability to determine if health interventions actually work in practice, and hence such trials can generate "real-world evidence" to inform implementation, clinical practice, and regulatory decision-making. 3,4 PCTs are "pragmatic" in that they are embedded in the workflows of diverse healthcare systems and aim to leverage existing data streams in the electronic health record (EHR) to limit the costs and burden of research data collection while maximizing the ability to answer important clinical questions and deliver better-quality care. 5 The Health Care Systems Research Collaboratory (hereafter, "Collaboratory") is supported by the National Institutes of Health (NIH) Common Fund and was designed to strengthen the national capacity to implement cost-effective, large-scale PCTs that are embedded in routine care and conducted in partnership healthcare systems. 6,7 The trials often involve cluster randomization (of hospitals, clinics, or primary care providers, etc.), the interventions may be implemented by health system personnel, and data are collected as part of routine clinical care. The Collaboratory is currently supporting 21 large-scale, high-impact PCTs (or Demonstration Projects) that address issues of major health importance, such as hospital-based infections, chronic pain, the opioid crisis, colorectal cancer, medication adherence, and suicide,  all in different phases of completion ( Table 1). The portfolio includes 6 new projects that are part of the Pragmatic and Implementation Studies for the Management of Pain to Reduce Opioid Prescribing (PRISM), which are part of the HEAL SM (Helping to End Addiction Long Term) Initiative. To apply lessons learned from ongoing projects and to provide expertise to the Demonstration Projects, 7 Core working groups were created: EHRs; phenotypes, data standards, and data quality; patientreported outcomes; healthcare systems interactions; ethics and regulatory; biostatistics and study design; and stakeholder engagement cores. A centralized Coordinating Center and the Core groups assist with the design, conduct, and logistical challenges of the projects and disseminate generalizable knowledge about PCT methods and enabling factors.
In 2017, the EHR Core reported the challenges from the Collaboratory's first round of 9 Demonstration Projects. Four broad informatics challenges were identified, including: "(1) using clinical data for research, (2) integrating data from heterogeneous systems, (3) using EHRs to support intervention delivery or health system change, and (4) assessing and improving data capture to define study populations and outcomes." 20,36 Using qualitative data generated from an electronic survey completed by lead scientists for each of the 21 PCT projects in the Collaboratory portfolio, we update and elaborate on the EHRrelated challenges. This collection of examples and prerequisites for successful ePCTs aim to improve the ability to use data collected during routine care to support research 37 and generate new knowledge in the form of implementable clinical interventions to address important public health issues.

Study design and population
This online cross-sectional and semistructured survey that included 4 open-ended questions was circulated to all 21 Collaboratory Demonstration Projects (including active and completed studies and 2 new projects in early design stage).  Collectively, these PCTs in the Collaboratory address a range of critical public health problems, including the opioid crisis, suicide, and colorectal cancer, using a variety of interventions, research designs, settings, and patient populations (Table 1).

Survey content and administration
The semistructured survey was created by members of the EHR Core and consisted of 20 questions, informed by known challenges. 20,36,37 The survey included 4 open-ended questions: (1) Is there EHR functionality that you would like to have for your study, but didn't? If yes, please explain. (2) Which EHR data elements are you using for your study? To the best of your knowledge, are they standardized across the EHR systems used in your study? (3) What key question(s) would you ask of EHR vendors to assess their readiness to support research? and (4) What data or functions would you like to see standardized across all EHR systems? Sixteen structured questions pertained to challenges relating specifically to data and data access issues, requiring either "yes" or "no" responses (Supplementary Table S1). If the participants answered "yes", then they were prompted to elaborate on their experiences. Completion of the survey was voluntary. Data were collected from Collaboratory demonstration project representatives between August 20, 2020, and June 30, 2021.

Data analysis
The responses from participants on the open-ended questions amassed a large amount of rich text, and we conducted a thematic analysis 38 on the resulting data. We coded narrative responses from the 4 open-ended questions directed to each project. First, RLR pre-   pared the data for coding by collating and unitizing the responses. Next, coding was completed by RLR to develop an initial codebook. A second coder (KSM) reviewed the initial coding schema and suggested changes as applicable. Group consensus was achieved for each coding unit, resulting in Cohen's kappa coefficient of 0.72. The resulting 21 codes were iteratively grouped by RLR and BJD into families, eventually resulting in the final 6 themes. This process is summarized in Figure 1.
The consensus themes and underlying codes were compiled and presented to the Collaboratory EHR Core for discussion between November 2020 and December 2020. During these meetings, the EHR Core members provided clarification on the 6 themes and discussed their implications.

Development of prerequisites to facilitate PCTs
The EHR Core used additional calls to brainstorm and discuss strategies and recommendations for enhancing EHR systems to support PCTs. We received input from our expert and diverse members of the EHR Core, Healthcare Systems Interactions Core, and Patient-Reported Outcomes Core. Through the circulation and collaborative writing of this report, members of the EHR, Patient-Reported Outcomes, and HCS Interactions Cores contributed to the list of prerequisites for PCTs and recommended EHR enhancements to improve capacity for pragmatic trials.

RESULTS
A total of 20 out of 21 (95%) Demonstration Projects completed the survey shown in Table 2.
The specific EHRs across study sites are presented in Table 3.
The responses to the open-ended questions included 82 unique ideas that were compiled into 21 consensus codes and further refined into 6 broad themes ( Figure 1).

Survey themes
Theme 1: Inadequate collection and integration of patient-centered data, including patient-reported outcomes, questionnaires, and documentation of services received Seven projects-particularly those researching management of pain or psychological trauma-voiced a need for patient-centered data within health systems and EHRs. These studies reported having to collect primary data that were expected to be readily available, including patient-reported outcomes (PROs), previously completed questionnaires, and advance care planning conversations. 16 This led to time-and monetary-consuming adjustments as illustrated by a project using Epic when they said, "We would have appreciated greater ease and flexibility of automatically responding to patientreported information, including PROs to match participants' needs. Automatically assigning the intervention to specific patients using Epic EHR logic and based on EHR data was clunky, time consuming, and expensive. We wound up using the MyChart questionnaire functionality, but this was cumbersome, and the user experience was not optimal." Another study reported that even if patient-centered data were collected, it was not done in a standardized manner across health systems, preventing the ready use of these data for multisite pragmatic studies. We hoped to rely on self-report/patient reported outcomes that occur during pediatric well visits. . . many of the items were different across systems and there was a great deal of missing data for these data at each health system.
Finally, while some PROs were collected, not all the PROs needed for a trial were: We had 8 patient reported outcomes (PROs) that we wanted patients to enter into the EHR. . . . Only 4 of the PROs were included in the EHR, the other 4 will be collected by RED-Cap.
Theme 2: Lack of structured research data collection, standard EHR functions, usable interfaces, and standard workflows Several respondents commented on their experiences with data collection in the EHR environment. Some reports were positive, as the EHR allowed the structured collection of research data, making it easier to collate, analyze, and share across study sites. One study reported that even when data collection was standardized and integrated into the EHR, there were still varying levels of missing data across sites, and another reported notable errors in the accuracy of structured data. 39 Several projects shared requirements for very specific EHR functionality (eg, prescribing educational videos to patients, notifications to detect changes in medication dosages, customized communications to patients though patient portals, provider alerts regarding state-specific requirements for life-sustaining treatment) that was not readily in place, necessitating the creation of new EHR-based tools or manual extraction of specific data. One respondent suggested that the variability in EHR functionality across sites can introduce potential usability concerns, potentially affect fidelity to the intervention, 40 and hinder scalability of clinical interventions.
We had to use different aspects of EHRs to implement our intervention. Some used the EHR with a static message, another the EHR with a dynamic "pop-up", one used the Radiology Information System (RIS)/dictation module. Making all of these approaches work smoothly was a challenge.
The EHRs we are working with have little customization capabilities for clinical decision support reminders and alerts. Two of them do not even allow clients to add any new reminders/alerts. Clients can only turn on/off reminder/alerts that are implemented by the vendor and those are often suboptimal. Also, the user interface of reminders/alerts is very basic and does not support actions such as placing an order.
Another project commented on scalability when they stated that, "Better integration of a user-centered interface that is scalable across health systems and EHR vendors [is important for data collection]." Middle-phase projects are enrolling and collecting data but early; complete or near complete are nearly done with enrollment. EHR: electronic health record.  x CompuGroup x EHR: electronic health record.
Theme 3: Lack of standardization of structured data for research Ten studies reported challenges with standardized data across sites. Studies had challenges with standard data elements (n ¼ 5), standard coding (n ¼ 2), inconsistent use of standard coding (n ¼ 2), and lack of standardized or structured (imaging) reports (n ¼ 1). Several studies reported that their study would benefit from the adoption and use of standard code systems-specifically for laboratory test results, procedures, medications, and indications for imaging procedures, even across health systems that use the same EHR product.
The codes are for the most part standardized (same codes used across sites), but their implementation is not. Some settings used ICD-procedure codes and others CPT for the same procedures. . . One site did not use CPT codes but instead their own proprietary procedure codes. Sites had different limits to the number of diagnosis codes reported for encounters and some sites did not identify primary diagnosis code. Encounter type codes were inconsistent across sites.
Others reported challenges not only in the adoption of standard code systems but also in the consistent use of them. The variation across sites in turn necessitated further processing-and significant resources and time-to aggregate and analyze the data.
We use several EHR data elements. However, they are not fully standardized across systems using same vendor-due to nuances of local build. This is rate limiting in terms of EHR research.
Several projects reported that standard data elements or patient screening and assessment tools would have been useful for their studies. In some cases, these measures exist and in other cases would require the medical and clinical practice communities to develop and advocate for their development and adoption as standard practice. Respondents from a pediatric-adolescent study reported that research and practice would benefit from: Well Child, Well Adolescent, Adult primary care visit templates using brief, validated, evidence-based screening and assessment tools, . . . including "gold-standard" behavioral health (mental health and alcohol and drug) measures and clinical decision response algorithms (eg, NIAAA or AUDIT alcohol screening, S2BI or CRAFFT substance use screeners for adolescents, PHQ-2 or 9 þ Columbia Suicide Screener, etc.).
The team also suggests we: "Work towards consensus on sets of standardized core clinical measures, perhaps endorsed by bodies such as AAP or AMA or USPSTF [American Academy of Pediatrics, American Medical Association, United States Preventative Services Task Force], perhaps with CMS [Centers for Medicare and Medicaid Services] input, that major health systems could get onboard with, and build them into the platforms." Theme 4: Lack of resources to support customization of EHR systems for PCTs Nine studies identified tools and functions to support research that would be useful for their studies and pragmatic intervention. These included the ability to order an intervention, ability for local customization of EHRs (presumably faster and cheaper than if the EHR company does the work), standardized patient summarizes, configuration of EHR for research (for provider alerts about potentially eligible subjects; tracking patients on protocols; patient portal enhancements, including personalized messaging and randomization; and for research).
Respondents were able to identify several ways that the health system could have supported the EHR configuration for their trial but did not for various reasons, including doubt that the health system had the available resources to do so.
There may be some tension between what helps with clinical research and facility whereas lots of (understandable) caution on clinical delivery system side to allow too much tinkering that may overburden clinical workflow, so I think improved integration of PROs would be terrific but know this can be seen as a double edged sword.
Similarly, another project stated: Our issues are less about Epic and its capabilities, and more about bandwidth of providers and patients and the supremacy of clinical needs over research needs.
A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority.
Our trial required extensive support from IT programmers and report writers to enable data collection, cleaning, and analysis. EHR vendors allow extensive end user modification and specification that causes end-user engagement in report generation.
Eleven studies reported challenges with IT staff turnover and getting local IT resources to support their project.
Skilled technical staff may leave for better positions-in our health systems or outside companies.
A common feature is that IT departments have demands on them for patient care and billing, and therefore allocating IT resources to research is often not a priority, and a request might take months or years to be started. These challenges seemed to reflect communication issues related to prioritization.
One of the biggest barriers we have experienced is the CMIO and high-level leadership has committed to working on this research study, but when the time comes to actually implement the [EHR alerts] there is a communication gap between the CMIO/Leadership and the IT analyst doing the work. . . .the person doing the work was not privy to all the previous conversations nor do they realize there is a specific timeline that needs to be followed. This has created some slight delays and a lot of additional and unnecessary conversations.
Another project stated, Time allocation/resources is very important. If a site has other pressing priorities its difficult sometimes to get an IT analyst dedicated to work on the research items even with a subcontract.
Theme 5: Difficulties aggregating multidata type resources for multisite trials While many trials were able to collect the data they needed, the aggregation of all the data from multiple sites presented as a common challenge. This presented as both challenges in sites that are not associated with the health system, and as data elements without common models or standards.
We are using pharmacy refill data to identify patients and deliver the intervention to patients. Pharmacy refill data comes from both internal pharmacy data as well as external pharmacies. We are currently able to get daily pharmacy refill data from health system internal pharmacies. For prescriptions filled pharmacies outside of the health system, we are not able to obtain daily pharmacy refill data currently.
Eleven studies faced challenges with the capacity to integrate their data with external sources and to accurately match data across these sources. Among these are challenges with integration with internal/external data sources, access to primary care provider practice data, identity matching across data sources, and lack of integration between care settings (specifically for dosage and medication changes).
[.challenges with]. . . Integration with specialty care, and procedures. Specifically access to colonoscopy outcomes and pathology lab data from biopsies. Claims data and cancer registry data would also been useful for exploratory outcomes. It would be useful to access data from [Epic's health information exchange module] Care Everywhere for research.
Data harmonization across different EHR products and even the same EHR at different health systems is very challenging and labor intensive.
Theme 6: Difficult and inefficient EHR data access: lack of reusable and automated queries to support PCTs Eight studies reported challenges with access to data for analysis. Several survey respondents expressed difficulties with automated data pulls from sites and also coordinating data across sites that update and send data at different intervals. These studies also suggested more widespread use of data warehouses and better access to data, with standard approaches to storing and retrieving data, including common data model (CDM) support.

Using the [site's] VDW [Virtual Data
Warehouse] was problematic due to variation in frequency of updates. Similarly, there were differential and sometimes substantial delays as to when EHR data became available.
. . .it would be extremely useful to be able to access medication dispensing records through our EHR data.
In addition to standardized/reusable data queries, we had requests for data cleaning support, data quality (including timeliness), and information around data provenance. It makes a difference if the data is coming from a data warehouse group that has a layer of management of those data rather than a pure export of existing fields. Understanding latency is important. There are also quality differences at the item level, and making that transparent is important.
Other studies reported challenges and lack of tools to pull data from the EHR during the study to support the intervention and conduct of the trials.
We wanted to automate data delivery from EHR systems in order to reduce burden on site research coordinators, but very few study sites used EHRs that were built with an enterprise data warehouse that would permit straightforward data pulls, so complicated workarounds, manual data transcription, and complex electronic reports had to be created.
We had wanted to use the RedCap integration with [EHR product], but it does not pass the Privacy and Safety requirements for 3 of our 4 sites, and for the 4th it won't pull custom data points from [EHR]. They are limited to pulling predetermined data elements.
Not all of our sites are able to download medication use and healthcare utilization data. One site, a Federally Qualified Health Center (FQHC), has an EHR with limited functionality. In addition, patients from the FQHC may be seen at local hospitals that are not part of the same health system, further complicating the automatic data extractions. Currently, automatic extraction is prohibited across systems via local agreements. Hence, we plan 'hand extractions' of EHR data.

PREREQUISITES FOR CONDUCTING PCTS
The EHR Core, PRO Core, and Health Care Systems Interactions (HCS) Core used the thematic EHR-related challenges reported in Collaboratory PCTs to identify prerequisites and recommendations for successful PCTs, which includes enhancing EHR systems to support PCTs. These are provided below.
Integrate the collection of patient-centered data, including PROs, questionnaires, and advance care plans, into EHR systems and clinical workflows to support pragmatic research and personalized clinical care Healthcare systems need incentives to invest in the routine collection of PROs, and EHR vendors and the Office of the National Coordinator (ONC) for Health Information Technology could better champion, require, enable, or support the collection of PROs. Potential benefits for integrating PROs into the EHR include increasing clinician/patient communication, 41,42 improving patient satisfaction with care, 42 and increasing symptom monitoring, which can improve clinical outcomes, such as survival. 43 In addition, participantprioritized outcomes (eg, anxiety, pain) might differ from traditional clinical outcomes 44 and help with risk prediction. 45 Research demonstrates that integration of PROs into the EHR is possible in a number of different types of care settings, 46,47 and facilitating additional measures for research is becoming more of a priority. [48][49][50] If PROs have not been established before a PCT is being conducted, clinics could potentially use the trial to help identify domains to include in their EHR, although there is great variability in what patients think is important depending on the disease or condition. However, integrating PROs into the EHR can change workflow and requires buy-in from multiple levels, [50][51][52] including input from clinicians and administrators at each site, as well as alignment with the priorities and the mission of healthcare system, and this is not trivial to accomplish. Finally, integration of PROs requires careful consideration and prioritization of what to measure and how to measure it, as sometimes PROs have not shown any effects/improvements in pragmatic settings. 53 Some groups have started to compile recommendations on how to successfully incorporate PROs into the EHR, 54,55 and a staged approach is often helpful. Ideally, systems should identify standard mechanisms through which PROs can be added and collected for specific projects, as adding too many PROs can add to burden in the clinical workflow, and it is hard to predict what PROs a future study might need.
Facilitate high-quality structured research data collection by leveraging standard EHR functions, usable interfaces, and standard workflows Principal Investigators (PIs) and their teams requested the ability for local customization of EHRs, standardized patient screening and assessments, and configuration of EHRs for research (eg, to provide alerts for potentially eligible subjects or providers, track patients on protocols, and patient portal enhancements, including personalized messaging and randomization, and for research). In many cases, the EHR functionality (eg, flagging potential or enrolled research subjects) that researchers requested was available in the EHR, suggest-ing that organizations were either unaware, under-resourced to implement them, or do not prioritize the use of EHR data in research.
Many of our challenges are not with the EHRs but with the health systems not using functionality. Although low-resourced settings might need a considerable amount of help to implement and maintain EHR functions, leveraging preloaded standardized EHR functions can accelerate implementation of innovative and transformative clinical practices and aid in generation of data from patient populations, which can contribute to improving the delivery of health care, as in a Learning Health System. 56 Ensure adequate IT and other staff or services to support embedded research The primary role of healthcare organizations is to provide the best patient care possible at the lowest cost, and conducting research is a secondary goal. Staff turnover has been consistently reported as a challenge for Collaboratory PCTs, particularly IT staff turnover. 57 All of these activities require local IT support, and local IT departments prioritize clinical care and billing, so obtaining IT support can be a challenge, a problem (understandably) exacerbated by the urgent needs created by COVID-19.
Devoting sufficient resources to IT support of PCTs is needed, and some members of our EHR Core suggested that research teams could hire IT staff solely dedicated to research; this could occur at the institutional level and be funded by research dollars. However, some of our investigators who have done this already report that the amount of funding is negligible next to the institution's bottom line, and this research-charge model might reduce the amount of organizational HIT funding (in response to transitory grant funding which eventually end), possibly acting as a disincentive. One hired personnel with experience with the most common EHRs and used them to work with FQHCs to implement and configure EHR functionality.
Prioritize the creation of structured research data by using and promoting standards Healthcare organizations could set priorities for IT staff so they increase use EHR data standards to support quality healthcare delivery, continuous quality improvement, and the generation of new knowledge. A main incentive comes from policymakers and regulations that affect payment, such as Uniform Data Systems (UDS) reporting for FQHCs and Meaningful Use requirements by the Centers for Medicare and Medicaid Services (CMS). 58,59 Relevant standards include standard data elements, standardized coding systems and terminology, or structured reports, any of which could be incorporated into common EHR builds. Health systems can request that vendors improve their systems, and perhaps the most efficient way to achieve this is to synergize efforts with agencies that impose requirements tied to healthcare payments, such as CMS/Healthcare Effectiveness Data and Information Set (HEDIS) quality measures and the United States Core Data for Interoperability (USCDI) data elements set by the ONC/CMS. 60 Create aggregate, multidata type resources for multisite trials Because our pragmatic trials were designed to minimize burden of de novo data collection, all make use of current data in the EHRs, but many required other data types and sources, such as commercial pharmacy data, practice data (on provider and practice features), and PROs, which needed to be accessed and linked. The aggregation of all the data from multiple sites presented a common challenge, due to both the fact that different systems might collect data using different data elements and have varying adoption of different CDMs. Both complicate the development of systems and interfaces for researchers to access the data required for multisite trials.

Create re-usable and automated queries to support PCTs
Respondents also reported the need to re-access data throughout the lifespan of their trials, and new queries, or queries of multiple sites, was required. A need for ongoing support for queries (or potentially an automated data pull) was voiced by the respondents. Data are configured differently for the "same" clinical uses across multiple sites, and this makes developing queries even more complex. EHR infrastructure should be designed so that data query and output is efficient and research friendly. If the future is about big data science and using existing data, then we have to a priori create a system that supports that vision.
As we noted in our previous publications, 20,36 the lack of standardized data and EHR functions means that programs for data queries, data management (including data quality assessment), study conduct (eg, automated alerts, notes, and dashboards to support the trial itself), and analysis need to be created for every health system participating in a trial. To overcome this need for custom programming at each site, Collaboratory-affiliated researchers recommended increased collaboration between clinical researcher informaticists and healthcare IT operations professionals, as has been previously described by the Collaboratory and others. 20,36,61 The establishment of multisite registries with data warehouses across health systems, especially for low-resourced settings, would help, and this is happening in several states. For example, in Utah, there is a data warehouse that is automating data feeds from 12 different health systems (FQHCs) using different EHRs. Another similar approach is the creation of a network that uses a CDM, such as the PCORnet, 62,63 although transforming data into a CDM can be time consuming and resource intensive. 64

DISCUSSION
Our findings add examples of specific services and standards, but the overarching question we face remains: how can researchers persuade health care organizations to invest in EHR enhancements and data infrastructure to support PCTs? Our findings from this survey of Collaboratory projects are similar to what we found in 2017: using EHR data for research and obtaining data from multiple sites are still major challenges; the amount of time and resources needed to use the EHR to support the delivery of interventions remains daunting, capturing high-quality data suitable for research is still difficult. While it might seem as if the needle has not moved on any of these issues, it is possible that studies are evolving in their data needs (eg, more complex interventions, more targeted and personalized interventions integrated with EHR systems) and that we are EHR-enabled to answer more sophisticated research questions. To better assess the improvement of these PCT facilitators at a national level, the Collaboratory Coordinating Center explores metrics for interoperability of EHRs and data "readiness" 65,66 for pragmatic research. A number of national developments indicated that standardization of EHRs and data are progressing. In addition to the recent ONC report (2020) promoting a research agenda for EHR-enabled research, the use of standards has certainly improved with Meaningful Use certification requirements, and the consolidation in the EHR market has reduced variation in EHR systems. Also, improvements in reporting infrastructure for quality measures have benefits for research.
Our findings also mirror areas of need in the 2020 report from the ONC, which calls for improvements in data quality, harmonization, access to interoperable data, services for efficient storage and data, integration with other health data sources, and data aggregation. 37 The use of EHR systems and data takes effort, costs money, and takes away from other health system priorities. This seems fundamentally unchangeable, so the alternative is to change the "story" and the argument for robust and customizable EHR systems. The story needs to not be about promoting research per se but rather about finding and implementing the best and most effective treatments, and continuously learning within and across health systems. However, building and promoting infrastructure for gathering standardized, re-usable data from the EHR in support of continual learning are not without costs, and it is the joint responsibility of funders, researchers, healthcare systems leaders, EHR vendors, and policymakers to work together to accomplish these goals. The National Academy of Medicine called for the development of a learning health system 11 years ago. It stated that 90% of clinical decisions should supported by timely, up-to-date clinical information, 67 but achieving that goal remains distant and will require "health system leaders to consider rigorous evidence generation a core function of ordinary health care, research funders to prioritize practical questions relevant to population health and to support infrastructure for embedded research." 56 Currently, patients bear the costs of a lack of evidence, and we must strive to change this. This might be facilitated using expertise and approaches from emerging Implementation Science, [68][69][70] which provides a scaffold for evaluation and bringing about change in the healthcare setting. In a recent article by Collaboratory leadership, the authors suggest re-imbursing healthcare systems for research-related costs and supporting re-useable infrastructure in highly engaged systems as a way to mitigate the costs of engaging in research and encouraging healthcare system participation. 71 Similarly, the inclusion of robust cost impact studies might demonstrate a business case for the use of effective treatments (and the deimplementation of ineffective ones). Collectively and over time, these studies might provide compelling justification for continuous pragmatic research in learning health systems.
Research stakeholders (including clinicians, policymakers, research sponsors, patients, and the public/healthy consumers) must persuasively promote the importance of high-quality data. This value is not limited to a specific trial, but extends to other uses such as population health, continuous quality improvement, comparative effectiveness research, pharmacovigilance, and even algorithmic and artificial intelligent safety surveillance in the future. These programs can be conducted more cheaply and effectively with high-quality data. Standards-based EHR systems can empower customers to adopt new practices, which can lead to improvement and implementation in health systems. However, the current status quo is for each instance of an EHR is to not be standardized, even across separate instances or Epic, and the many different data sources (16 across the 21 Collaboratory trials) creates an ongoing challenge for investigators conducting PCTs (ie, for any individual trial, this means that every data pull needs to be configured for each and every participating site). The value of standards-based EHR systems can be promoted to health system leaders as a necessary enabler for integrating evidence-based practices and intervention to improve care and the health of the populations served (ie, a means to link research to quality improvement and continuous learning).
A positive and hopeful development is the HEAL (Helping to End Addiction Long Term) studies in response to the opioid crisis. These 6 studies are supported by the Collaboratory and have helped define a set of common measures 72,73 based on research and clinical needs, including a common set of PRO measures. Using this common set of PRO measures will help with comparability across trials, minimize patient burden, as well as provide valid and reliable quality-of-life measures. 74 One next step may be to lobby EHR vendors to include these basic measures and for EHR regulators/standards bodies to endorse them. Should the interventions evaluated by these studies (acupuncture, physical therapy, TENS, guided relaxation group-based mindfulness, etc.) be effective, the use of consistent measures across these studies could help with broader implementation. More recent success with N3C and COVID have shown that with the right incentive (and a bolus of centralized resources for harmonization), investigators can pull together disparate data quickly to understand the disease burden and effective targets for limited resources. 75 Our study teams also reported the need for configurable EHR systems. Often informaticists tightly control the EHR in an organization, and the key stakeholders, including policymakers, are not fully aware, empowered, or incentivized to champion or facilitate the EHR customizations needed for multisite trials. Enhancing the ability to customize EHRs enables responses to urgent public health needs, delivery of new and evidence-based treatments, and the ability to address emerging local needs and patient-specific care. The number of PCTs funded by NIH and supported by health systems is growing; implementation science has burgeoned as a field; research networks, such as the Patient-Centered Outcomes Research Network (PCORnet) are enabling use of harmonized data; and the FDA is championing the use of real-world evidence. At the same time, the humanitarian and financial issues of poor health metrics, disparities in health, and increases in chronic and multiple diseases are growing. We have not made a dent in the Triple Aim, and the Quadruple aim, [76][77][78] which adds the dimension of preventing clinician burnout and frustration with EHR systems.
Driving change for routine PCTs will require meaningful partnerships between healthcare systems, individual clinics, informatics teams, EHR vendors, and researchers, as well as federally driven standards and policies to ensure that EHR systems can support research. 79 Indeed, the 21st Century Cures ACT 80 and other developments are laying the case for this, and we assert that clinical researchers and health services researchers-as representatives of their health delivery organizations-should be leading efforts to develop and promote data standards that can enable sharing and rapid adoption of tools that facilitate the trial. These researchers can also leverage new regulations for quality measures, and weigh in and advocate for the importance of PROs. Embedded PCTs are integral to a learning health system and will enable faster changes in practice and improved patient outcomes. In time, the case for standards will grow in clarity and support as the learning cycles get faster and the benefits become more visible and established.

Limitations
Our sample of studies is limited and may not represent all pragmatic trials, and our sample varies in terms of timing of research, from beginning of enrollment to complete, and was conducted during the COVID-19 pandemic, which may have exacerbated issues related to EHR use and enhancements. The types of studies in our sample vary greatly, as the Collaboratory is funded by the NIH Common Fund with the goal of learning how to conduct high-impact trials that matter to public health, although this variation may not be representative of all PCTs and may affect the conclusions we can draw about EHR system readiness for research. Another limitation is that responses were collected in an asynchronous survey, so classic interview methods could not be done; that is, responses could not be expanded upon.

CONCLUSIONS
The case for high-quality, standardized data goes beyond clinical care, as it can impact the population as a whole by providing better care based on reliable evidence. We argue for tailoring the use of EHR systems to enable the collection of patient perspectives and the extraction of high-quality, robust data to support pragmatic research toward the identification of effective treatments and implementation strategies, and to enable learning within and across healthcare systems.

AUTHOR CONTRIBUTIONS
A study representative and PI reviewed the final summary of each study and its features presented here. All authors listed here participated in the discussion and refinement of the themes and suggestions we present here and provided substantive comments on the manuscript.

SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.