{'fraction_non_alphanumeric': 0.018952339526502205, 'fraction_numerical': 0.006648853538805692, 'mean_word_length': 5.257192846034215, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A narrative review of data collection and analysis guidelines for comparative effectiveness research in chronic pain using patient-reported outcomes and electronic health records

Chronic pain is a widespread and complex set of conditions that are often difficult and expensive to treat. Comparative effectiveness research (CER) is an evolving research method that is useful in determining which treatments are most effective for medical conditions such as chronic pain. An underutilized mechanism for conducting CER in pain medicine involves combining patient-reported outcomes (PROs) with electronic health records (EHRs). Patient-reported pain and mental and physical health outcomes are increasingly collected during clinic visits, and these data can be linked to EHR data that are relevant to the treatment of a patient’s pain, such as diagnoses, medications ordered, and medical comorbidities. When aggregated, this information forms a data repository that can be used for high-quality CER. This review provides a blueprint for conducting CER using PROs combined with EHRs. As an example, the University of Pittsburgh’s patient outcomes repository for treatment is described. This system includes PROs collected via the Collaborative Health Outcomes Information Registry software and cross-linked data from the University of Pittsburgh Medical Center EHR. The requirements, best practice guidelines, statistical considerations, and caveats for performing CER with this type of data repository are also discussed.


Introduction
The Institute of Medicine (IoM) estimates that chronic pain affects over 100 million people in the USA and is associated with over $600 billion in health care costs. 1 Chronic pain is a multifactorial condition that is notoriously difficult to treat effectively. The specific pain and symptoms experienced by each patient are the result of complex interactions between their anatomy, physiology, and biopsychosocial environment. 2 Although generalized treatment algorithms have proven effective for diseases like hypertension and diabetes mellitus, there are few and possibly no reliable and validated treatment algorithms for chronic pain, and pain medicine practitioners often need to develop individualized care plans for their patients. They must take into consideration the possible responses to a whole range of treatments such as medications, injections, rehabilitation, and mental health care. Thus, knowledge about the real-world effectiveness of treatments for chronic pain is crucial to advancing the field and improving patient care.
In the landmark report, Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research, the IoM (now known as the National Dovepress Dovepress 492 Dressler et al Academy of Medicine) laid out guidelines for conducting comparative effectiveness research (CER) to develop treatment regimens for chronic pain that optimize benefits to the patients while minimizing the cost of care. 1 CER is a clinical research method in which established medical interventions are directly compared to determine which are the most beneficial and cost-effective. 3 The IoM also promoted creating Learning Healthcare Systems where clinical practice is informed by scientific evidence, and reciprocally, clinical research agendas are influenced and informed by patients and providers.
The collection and integration of patient-reported outcomes (PROs) are essential to the Learning Healthcare System model. While electronic health records (EHR-also called electronic medical records) provide a wealth of information about the demographics, diagnoses, and treatments administered to each patient, PROs provide a quantitative measure of how those treatments affect each patient. Combining EHRs with PROs allows researchers and clinicians to perform CER at an unprecedented level of coverage and precision. 4 The Collaborative Health Outcomes Information Registry (CHOIR) is a PRO tracking system designed specifically for chronic pain patients and pain management programs. 5 CHOIR is an open-source, web-based software program modifiable at each local installation that was developed by researchers at Stanford University. When the PROs are combined with data extracted from the EHR, this repository provides a powerful tool for CER in chronic pain. This review will outline best practices for conducting CER using PROs and EHRs using the CHOIR system as an example. While this paper specifically focuses on chronic pain and pain management programs, these guidelines are applicable to any complex medical condition and any type of health care system in the USA. ceR using practice-based evidence CER can be performed with a variety of clinical study designs. Randomized controlled trials (RCTs) have long been considered the "gold standard" of exploratory clinical research, but these studies require significant amount of time, money, and researchers committed to each project. RCTs are also very inefficient; the IoM estimates that it takes 17 years for new knowledge derived from an exploratory RCT to be incorporated into clinical practice. 6 Observational study designs provide an alternative to RCTs that allow researchers to examine patient populations without manipulating their treatment regimens. While observational studies are usually cheaper and more efficient than RCTs, the quality of these studies is often affected by patient and treatment selection bias, confounding variables, and the use of a heterogeneous subject population. 7,8 Practice-based evidence (PBE, also called Clinical Practice Improvement Methodology) is a model of CER that is designed to minimize common biases in observational studies. In a 2012 paper, Horn et al outlined the requirements for rigorous PBE studies. 8 In these studies, multidisciplinary and multisite project clinical teams utilize comprehensive age-and disease-specific measures that are implemented as widely as possible and checked regularly for reliability. PBE data are merged into a central study database for analysis and hypothesis testing. Study findings are then implemented into clinical practice for validation testing with the ultimate goal of integration into standard care. PBE studies are designed to improve on traditional observational studies by 1) examining large, diverse patient populations; 2) involving clinicians in the research design and data collection; 3) using carefully selected patient characteristics for analysis to avoid bias; and 4) standardizing data collection and treatment documentation at all research sites. PBE methodology is ideal for conducting "pragmatic" trials that are designed to measure the overall benefit produced by a treatment in a naturalistic clinical setting. 9 Reviews in the pain medicine literature indicate that such a pragmatic approach based on real-world clinical data may yield CER that can truly improve the treatment of chronic pain. 9 A PRO database combined with EHRs provides an ideal mechanism for conducting PBE studies and pragmatic trials. The outcomes measured can include quantitative measures such as pain intensity and global health scores as well as qualitative measures like the patient's impression of change. If outcome measures are collected routinely in clinical practice, researchers are then able to study a large proportion of the clinic population. However, establishing this type of database for PBE research is no small feat and requires a number of considerations for both the EHR and the PRO databases.

eHRs in ceR
The Health Information Technology for Electronic and Clinical Health Act was enacted in 2009 as part of the American Recovery and Reinvestment Act to initiate the improvement and implementation of EHR in a wider range of clinical settings. 10,11 While this law has led to greater availability of clinical and administrative data, there are a number of considerations and caveats for combining EHR data with PROs. Hersh et al provided a thorough review of the issues that can arise when using EHR for CER in a 2013 paper. 12 In short, submit your manuscript | www.dovepress.com

493
Dressler et al EHR data can be incomplete, inaccurate, and inconsistent. Enterprise data collection and data entry standards can vary greatly both within and between health care systems. It is critical for researchers to understand what data are included in EHRs, to carefully select variables for analysis, and to clearly report the limitations of their analyses as well as any potential problems with their datasets.
The most important consideration when using EHR for any research purpose is that the EHR contains identifiable patient data, including names, dates of birth, medical record numbers (MRNs), addresses, phone numbers, and insurance information, and must be stored according to the Health Insurance Portability and Accountability Act (HIPAA) guidelines. Researchers wishing to use EHR must either 1) obtain informed consent from each patient whose data are included in an analysis, which is often impractical, or 2) obtain a waiver of consent from an Institutional Review Board or research oversight committee. A waiver of consent typically requires that either the research staff have access to EHR data as part of their regular clinical duties or researchers only view data that have been deidentified by an honest broker. 13,14 Accessing EHR data for research purposes can also be a cumbersome task. Most institutions have a research office or committee that can assist with EHR data queries. However, the availability of resources for vetting project requests and obtaining EHR data vary greatly among institutions, which can lead to very long wait times for research data. To mitigate these issues, comparative effectiveness studies that require EHR data should account for when, how, and what EHR data will be acquired during the initial study design phase.

PROs in pain medicine
As stated above, the IoM's 2011 report emphasized the need to collect PROs for CER. 1 PRO is a catchall term to describe any health-related data provided by the patient, including health and treatment history, symptoms, and lifestyle choices. 15 In 2003, the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) recommended that clinical trials of chronic pain measure six core outcome domains to determine the efficacy and effectiveness of a treatment: 1) Pain, 2) Physical Functioning, 3) Emotional Functioning, 4) participant ratings of Global Improvement, 5) Symptoms and Adverse Events, and 6) Participant Disposition, which includes adherence to the treatment regimen. 16 A comprehensive PRO system should include measures that cover all six IMMPACT domains, but there is not an accepted standard currently for which specific measures should be utilized. 17 While a number of measures and guidelines exist, the specific makeup of a PRO questionnaire can be decided by clinicians, researchers, and/or administrators at each site.
The NIH Patient-Reported Outcomes Measurement Information System (PROMIS) was created in 2004 as part of the NIH Roadmap Initiative. 18,19 The PROMIS initiative established a collaborative network of clinicians, researchers, and measurement experts tasked with developing and validating item banks to measure symptoms and health statuses for a wide range of chronic conditions, including chronic pain. The PROMIS is currently administered by the NIH Health-Measures (also known as the Person-Centered Assessment Resource) grant, and all measures are available to clinicians and researchers free of charge at www.healthmeasures.net. As of 2018, there are over 300 PROMIS measures covering physical, mental, and social health that are applicable to both the general population and patients with chronic health conditions. PROMIS measures are available as full item banks, computer adaptive tests (CATs), fixed-length short forms, and comprehensive profiles. One advantage of the PROMIS measures is that the percentile scoring is benchmarked to US national population norms, including both diseased and healthy normal individuals. This facilitates comparisons across geographic regions and different health care populations.
For pain medicine CER, the PROMIS measures provide an ideal and effective way to collect PROs in chronic pain patients that cover all six IMMPACT domains (Table 1). Additionally, there are other validated measures available that provide an alternative and/or supplement to the PROMIS measures, including the Brief Pain Inventory, 20 the Oswestry Disability Index, 21 and the PainDETECT Questionnaire for Neuropathic Pain. 22 When choosing which measures to include in a PRO database, it is important to balance the breadth and quality of data collected with the burden on the patient. For example, PROMIS CATs use item response theory, 23 so the exact questions administered to each patient depend on that patient's previous answers and the total number of questions administered to each patient can vary from 4 to 12. Fixed-length short forms provide an alternative to the CATs, so each patient answers the same number of questions. Although this can reduce the maximum number of questions that a patient could be given, T-scores calculated from short forms may not be as precise as those from CATs. 18,19 For a PRO system to be effective for CER and useful in clinical practice, it is critical that 1) the collection of outcome data does not interfere with normal clinical operations, 2) the specific measures used are of interest to clinicians and researchers, and 3) the patients are engaged with and not

494
Dressler et al unduly burdened by the questionnaires. It has been wellestablished that as questionnaire length increases, both the rate and quality of responses will decrease. 24,25 While it is tempting to collect a wide variety of data to assess every variable of interest, it is important to understand that both data quality and response rates will be optimized when the questionnaire length is kept to a minimum. Patient burden can also be reduced by increasing the accessibility of the questionnaires, 26 by offering multiple options for survey completion (such as via e-mail or text messaging), offering surveys in multiple languages, and providing options for patients who are vision impaired, have low literacy levels, or are unable to use a computer or touchscreen. 27 A PRO system requires both motivated patients and motivated providers to ensure its success. The patient must feel that the data they supply at each appointment are important to their care and are actually being used by their provider. If a patient is able to see that their survey information is being used to directly guide their care, they are more likely to complete the survey and provide high-quality responses to each question. 25 This approach has also been noted to improve patient satisfaction with care scores. 28 It is important for providers to acknowledge and thank the patient for completing the survey and, if needed, to go over individual scores and answers. However, a potential downside of this enhanced patient engagement approach is that patients may be motivated to exaggerate their improvement to please their physician or exaggerate their deterioration to have additional treatments prescribed. The field is lacking good studies on the potential bias due to social desirability in completing PROs. Nevertheless, the ultimate goal when collecting PROs is to continually involve the patients in their own care and to take their needs, preferences, and priorities into account when making shared treatment decisions.

Patient outcomes repository for treatment (PORt) at the University of Pittsburgh
The data repository system at the University of Pittsburgh Medical Center (UPMC) joins PRO collection via CHOIR with extraction and crossmatching of individual patient-level data from the EHR. CHOIR is an open-source, web-based PRO software specifically designed for pain medicine programs, but applicable to a range of disease areas. 5,29-32 The CHOIR system includes PROMIS CATs and short forms, other validated measures, and site-specific customizable surveys.

Data collection
CHOIR outcomes are collected via surveys that patients are asked to complete at each clinic visit. To increase completion

495
Dressler et al rates and accessibility, patients are offered three options for survey completion. First, a few days before their scheduled appointment, patients are sent an email that contains a link to that appointment's CHOIR survey. Since CHOIR is entirely web based, patients can complete the survey before their scheduled appointment on their own computers or mobile devices. Second, if the patients do not complete the survey before their appointment, they have the option to complete the survey on a clinic-owned tablet in the waiting room. Third, patients who do not wish to use a tablet have the option of completing the surveys on paper. It should be noted that paper versions of CHOIR surveys need to be manually entered into the CHOIR database by clinic staff and need to be stored according to HIPAA guidelines.
It is common for patients receiving injections to return to a pain clinic after the injection has worn off. In such cases, the surveys collected at the patient's injection and follow-up appointments may show the same pain levels. Although the patient may have gotten significant relief from the injection, the pain scores collected at each appointment would suggest that the injection was not effective. To address this, a very brief survey (taking <2 minutes to complete) is emailed to patients every 6 weeks if they do not have a follow-up appointment within this time frame. Collecting these nonappointment follow-up surveys enables CER by obtaining PROs more frequently to improve the analyses of treatment effectiveness. A weblink to contact the office with any questions or concerns is also included with the nonappointment survey request. This feature enables remote patient monitoring to assist in population-based chronic disease management. 33 The CHOIR software was designed to be flexible so that each site could customize its questionnaires. Early versions of CHOIR allowed for two survey types: initial surveys for new patients and returning patients answering their first CHOIR survey and shorter follow-up surveys for all other appointments. Newer versions of the CHOIR software can create customized surveys for different appointment types such as procedures and psychological evaluations. In 2016, UPMC Pain Medicine implemented the CHOIR system into its eight regional clinics in western Pennsylvania. Using the flexible software architecture, UPMC customized initial and follow-up surveys to include PROMIS measures and survey questions from the original Stanford CHOIR surveys plus new measures and custom site-specific questions. Measures used in the UPMC version of the CHOIR system are italicized in Table 1, and the UPMC data dictionary does not include all of the PROMIS item banks described. More specifically, the UPMC CHOIR surveys have added the PainDETECT neuropathic pain survey, 22 a validated Opioid Adherence Checklist to identify self-reported misuse, 34 and two custom 5-point Likert scale "Impression of Change" questions (similar to the Patient Global Impression of Change scale) 35 that ask the patient how much improvement they have noticed since they first started coming to the pain clinic and for the most recent treatment they have received. The Impression of Change measure is a valuable tool because it adds a subjective patient-reported assessment alongside the more objective validated PROMIS measures. This is valuable, for example, in a patient who had an intervention and benefited from it and is now returning to the clinic because the previously beneficial effects of the intervention have waned. If a clinician or researcher was to look only at PROMIS pain intensity scores for that patient, it would appear that the patient's pain remained the same and there was no improvement. However, that same patient may report that they have seen meaningful improvement since they started coming to the pain center.
One concern often raised when gathering PROs is whether the data are more valid if the patients themselves complete the surveys without any assistance. However, a recent study has indicated that proxy-completed PROMIS measures are highly reliable in comparison with the patient-completed measures. 36

clinical uses of cHOiR
When patients complete a CHOIR survey in the waiting room or while they are waiting for the physician to come into the examination room, their answers are used to generate a custom PDF report for their providers. These can be reviewed prior to seeing the patient and can be reviewed with patients in the examination room ( Figure 1). These reports contain an image of the body map showing where the patient is experiencing pain, PROMIS T-scores, calculated scores from other measures, and a list of each question given and the patient's corresponding answers. Follow-up survey reports contain line graphs that plot the change in scores over time. This allows clinicians to easily see how the patient's pain, mental, and physical health have changed over time and in response to different treatments. This PDF report can then be pasted into the encounter notes in the patient's EHR. Future releases of EPIC (the EHR at UPMC and many other institutions) will enable the discrete variable data from the PROs and the PDF reports to be automatically transmitted from the CHOIR database to EPIC. The presentation format of the data to patient and provider is frequently reviewed through qualitative analysis of periodic semistructured interviews to determine if the clinical information is conveyed effectively. For example, in

496
Dressler et al the future, a more refined "dashboard" presentation with bar graphs may be more effective than the existing PDF reports.
A key strength of the CHOIR system is that data are directly reported and entered by the patients themselves and not based on or subject to the physician's report or documentation. This avoids interpretation or reporting bias that may be present in other systems. More importantly, this serves to integrate the patient into their own treatment team, which can strengthen the bond and enhance trust between patient and providers. This is particularly true when the clinician emphasizes to the patient that they reviewed their CHOIR report. This is an invaluable benefit of the CHOIR system, as the doctor-patient relationship can be difficult to establish in pain management because patients have often seen multiple specialists and had numerous unsuccessful treatments before they visit a pain medicine physician.

PORt database architecture
All CHOIR data collected from clinic patients are stored in a custom PRO database managed by the organization's Information Services Division. This database is stored and managed separately from the outpatient EHR database.
Patient data in the CHOIR database can be linked to EHR data by matching MRNs, encounter dates, and provider information. The CHOIR software contains a suite of tools to monitor data quality, including the number of completed and partially completed surveys.
Access to the outpatient EHR database is set up with a customized view filter so that all relevant EHR data (and no unnecessary data) are available to researchers and clinicians. The available EHR data include 1) encounter information for all pain medicine clinic patients at pain medicine clinic appointments and medical history information for these patients, including 2) medications ordered, 3) procedures ordered, 4) medical and surgical history, and 5) medical problem list. The same functionality applies to the inpatient EHR database at UPMC, which can be cross-linked to visit specific data from the outpatient EHR and the CHOIR PROs. Thus, health care utilization information, such as emergency department visits or hospital admissions, is available in a robust fashion. At the University of Pittsburgh and UPMC, this combined PRO and EHR data repository is referred to as the PORT. Figure 2 illustrates the processes required to generate predictive analytics from PROs and EHR data. Inputs of data are performed by patients, providers, and health care staff. Examples of data that can be collected via PRO and EHR are listed in Figure 2. PRO and EHR data are then fed into a shared database and linked via common patient identifiers such as MRNs. Data queried from the linked database must undergo some form of data processing to prepare for analysis. This data processing can include deidentification, categorization of variables (such as ages converted to age ranges), case-control matching, and/or feature selection to remove irrelevant and redundant variables from the model. Variables with predictive validity are then analyzed as covariates in the final predictive data analysis. Data processing should also identify the outcome variables to be used in the predictive data analysis, such as responses to treatments, clinically significant improvement in pain and/or function, or tapering of opioid medication.
At UPMC, the CHOIR PROs and outpatient and inpatient EHR information are stored in Oracle servers and comprise the PORT data repository. Data queries must be programmed in SQL and performed with SQL-based database software such as SQLDeveloper. This requires specific coding skills that may be difficult for physicians and researchers who do not have backgrounds in programming or data analysis. To address this, our research group at UPMC and the University of Pittsburgh has developed a data request form that allows clinicians and researchers to receive CHOIR and EHR data without needing database access or specialized programming skills. Data requests are vetted and fulfilled by research administrators and database managers.

Research with PORt
The PORT platform includes the integration of the CHOIR system into pain medicine clinics and linkage of its database to outpatient and inpatient EHR data. It provides an ideal data source to conduct high-quality practice-based CER in a Learning Healthcare System. With the PROMIS measures included in CHOIR and the customized patient PDF report for each survey, clinicians and researchers are able to evaluate each patient with tools that are both normalized and validated for the US population. Continuous collection of these measures at each appointment greatly facilitates personalization of treatment plans and evaluation of prior interventions. As of this publication date, we have collected over 60,000 surveys in >24,0000 unique patients, with each patient's PRO data cross-linked to their corresponding EHR data. On average, CHOIR data are captured in 70% of our patients. New surveys are accrued at a rate of approximately 500 per week.
There are many considerations regarding statistical analysis of PRO and EHR data for CER. Overall, CER must be based on valid data and sound analyses. Before data analysis can occur, researchers must select appropriate sample sizes and analysis methods for the research question, address the psychometric properties of the PROs, and perform data cleaning, formatting, and harmonization where necessary. 37,38 Making causal inferences from observational data, even when gathered in a high quality and consistent fashion, is plagued by a myriad of unmeasured confounders. 39 One of the most important is "confounding by indication," which refers to the nonrandom choice of the specific treatment by the provider. In other words, providers prescribe a certain treatment for a patient based on their unique presenting characteristics, which may not be measured or cannot be captured by the PRO or EHR data used for analysis. This introduces an inherent level of bias to CER because treatments are not being compared with each other between patient groups whose characteristics are equal (apples to oranges). 40 Multiple statistical procedures exist to minimize the effects of this confounder and many other unmeasured confounders. Some of the most recommended are multivariate regression modeling and using propensity scores. 41 Propensity scores narrow the patient populations for the treatments being compared statistically to those with comparable characteristics at baseline. 42 However, the variables used for matching at baseline cannot also be outcomes of interest (such as pain intensity levels), and thus, successful propensity score matching will not create groups of patients that have identical baseline characteristics as a whole. Least absolute shrinkage and selection operator variable selection methods comparing the quartiles of propensity scores between groups and using multiple propensity scores can improve the performance of propensity score matching in the causal inference models. [43][44][45] In particular, multiple propensity scores have shown to be effective in comparing multimodal combinations of treatments in musculoskeletal disease. 44 Research with cHOiR The CHOIR system has already been used to conduct several innovative research studies. Researchers at Stanford, where the CHOIR software was developed, have published at least 15 articles to date that utilized CHOIR data. Most of these studies used cross-sectional path modeling to examine the relationships between several PROMIS pain, mental, and physical health measures. These studies found that fatigue is a key mediator of physical function and pain interference 5 and that satisfaction with social roles is an important driver of the relationships between physical function, depression, anger, and pain. 29 Another cross-sectional study showed that perceived injustice has a strong correlation with anger in chronic pain patients, and this relationship is at least partially mediated by social satisfaction. 30 Perceived injustice also appears to contribute to greater use of prescription opioids, increased pain intensity, and decreased physical function and pain acceptance. 32,46 Using similar methods, researchers also found that prescription opioid use is related to pain catastrophizing and pain intensity, and at low pain catastrophizing levels, women are more likely than men to have an opioid prescription. 47 Stanford researchers and clinicians have also developed a Pediatric-Collaborative Health Outcomes Information Registry (Peds-CHOIR) to improve treatment and facilitative research in children with chronic pain. 31 Examining both adult and pediatric chronic pain populations has allowed researchers to discover how chronic pain changes with age. For example, one cross-sectional modeling study found that the relationship between pain catastrophizing and pain interference is strongest in adolescents but decreases with age. 48 These groundbreaking CHOIR research studies conducted in active pain medicine clinics have allowed clinicians and researchers to examine, model, and understand chronic pain in ways that are simply not possible with traditional clinical research studies. However, there has not been a great deal of CER conducted using CHOIR data. This paper has outlined why the CHOIR PROs system combined with EHRs provide an excellent tool for CER in pain medicine. By examining existing patient populations and tracking how they respond to their treatments, clinicians and researchers can answer many "big" questions about pain, such as which treatments work for which patients, which treatments do more harm than good, and how do we define when a patient is "better"?

Conclusion
The combination of PROs and electronic health data is an innovative and exciting mechanism to conduct CER in chronic pain. These tools will allow pain medicine providers to more easily and accurately develop individualized treatment plans for their patients. Patients, in turn, will be empowered to be directly involved in their own care and to take ownership of treatment decisions. This can improve the patient-provider relationship and ultimately lead to better and more effective care in pain medicine.