{'fraction_non_alphanumeric': 0.03066502463054187, 'fraction_numerical': 0.020073891625615764, 'mean_word_length': 5.142586989409985, 'pattern_counts': {'":': 0, '<': 6, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Validation of an automated mortality index using the electronic medical record system in a network of acute care hospitals

Objective: Physicians struggle with prognostication for patients facing the final year of life. Practical tools which identify patients at the time of hospital admission who are at high risk of mortality would be helpful to provide timely access to supportive services, including palliative care and hospice. The PREDICT is a validated tool that predicts mortality risk but has not been implemented into electronic medical record (EMR) systems. The current study evaluated the validity of PREDICT within an EMR system and tracked patient mortality over 12 months. Methods: The study sample consisted of 3,488 adult patients admitted to a network of acute care hospitals. The PREDICT tool was evaluated for its ability to predict mortality within 6 and 12 months of hospitalization and was compared to the APR-DRG Mortality Risk Index (MRI). Results: A total of 299 patients (9%) were deceased within 12 months of hospital admission. Logistic regressions revealed that higher PREDICT scores were associated with greater risk of mortality within 6 and 12 months post-discharge. Receiver Operating Characteristic curve (ROC) analysis revealed that the overall PREDICT score significantly predicted mortality at 12 months (ROC = .767) and was a better predictor than the MRI. Conclusion: The PREDICT tool is a valid assessment of mortality risk and unlike the MRI, it can be readily automated in the EMR to help identify patients at greater risk of death. More research is needed to apply this tool in clinical practice and calibrate its performance across clinical settings.


INTRODUCTION
Identifying patients at risk for mortality is a challenging task for clinicians and ultimately impacts both the patient and their families. Prognostication, particularly for patients with comorbidities facing their final year of life, is difficult. [1] Pa-tients and families often do not receive desired information about end of life options and access to supportive services including palliative care or hospice. [2] Existing mortality risk tools exist but are often time-consuming to use in clinical practice or are disease specific and not applicable to a broader range of patients with multiple comorbidities. [3] Further, few tools address the final year of life and instead address mortality risk over many years. [4] With the widespread use of the electronic medical record (EMR) system in healthcare, providers now have the unique opportunity to access readily available patient health information to help inform and assist with prognostication.
Despite the advances in access to patient information, mortality risk tools exist but are often dependent on administrative claims data (i.e., All Patient Refined Diagnosis Related Groups APR-DRG Mortality Risk Index), and have not been used for real time clinical decision support. [9] The tools that have been developed to assist clinicians in prognostication for 1 year mortality are restricted to the nursing home setting, specific populations, or the community setting. [4][5][6] Additionally, the tools that utilize hospital admission information rely on provider assessment of functional status which is not routinely done in clinical practice, or data entry for calculations; all of which require time and effort on the part of the clinician. [4,7,8] Clinicians need a validated tool that is EMR-driven and addresses the most pertinent needs of the patient in their final year of life.
The PREDICT tool was developed to identify patients who have a high likelihood of death within one year of hospitalization, using information readily available at the time of admission. [9,10] Six criteria inform the overall score including; recurrent hospitalizations, prior cancer diagnosis, ICU admissions with multi-organ system failure, nursing home residence, age, and prior use of palliative care. [11,12] This tool has been validated in retrospective form through chart abstraction and was found to have specificity of over 90% and sensitivity of 33%-54% in multiple studies. [13,14] Despite this work, the prognostic tool has not been implemented into the EMR for real-time decision support to aid clinicians in prognostication. Such an implementation would enable providers to communicate earlier goals of care with patients and families and allow for prompt access to hospice and palliative care services. [11][12][13][14] We created the first automated PREDICT tool within our EMR system to help clinicians across a network of hospitals identify patients with advanced serious illnesses who may benefit from additional services and support. The current study aimed to validate the automated EMR version of the PREDICT tool in a population of adult patients hospitalized across a network of acute care hospitals, to compare PREDICT to an existing mortality risk tool, and to determine if this tool could inform risk of rehospitalization. A validated PREDICT tool in automated form would broaden its applicability to the design of clinical decision support interventions within the EMR, and could be scalable to other hospital systems.

Design and setting
This study was an observational cohort of adult patients (age > 18 years) admitted to eight acute care hospitals in the Southeastern United States from September 23, 2016, through October 23, 2016. Patients admitted to psychiatry, pediatrics, obstetrics, endoscopy, or procedural suites were not included in the cohort as the tool was originally created for adult patients at time of acute hospital admission. In the event of multiple hospitalizations for one patient during the study period, only the initial index admission was included. The study was approved by the local Institutional Review Board as non-human research, thus informed consent was waived.

Population
A total of 3,488 unique inpatient admissions met inclusion criteria. Patients were female (53.0%), primarily Caucasian (58.1%) and African American (34.7%), and had a median age of 63 years.

Data source and variables
All variables were extracted from the EPIC EMR and deidentified for analysis.

Demographic and clinical variables
Study variables included age, gender, ethnicity, insurance status, comorbidities, presence of an advance care plan, PRE-DICT variables, [12][13][14] rehospitalizations within 1 month and 6 months, and mortality within 6 and 12 months. Additionally, the APR-DRG Mortality Risk Index was also included in the study for comparison since it is a validated, proprietary clinical model used nationally for quality measures and reimbursement. [9] The primary outcomes of interest were mortality within 6 and 12 months of hospitalization and rehospitalization within 1 and 6 months of discharge. To ensure accuracy of mortality data, mortality was determined using multiple sources including hospital status death within the EMR, and a review of the National Technical Information Service (NTIS) Death Master File.

PREDICT tool
The PREDICT tool was developed to asses risk of mortality within 1 year. [12] PREDICT yields a numeric overall score ranging from 0 to 41, and associated categories indicating low (score < 4), medium (score between 5 and 12), and high (score > 13) risk of mortality. Both the total PREDICT score and the associated PREDICT category (low, medium, high) for each patient visit were extracted from the EMR (EPIC) and used for analysis.

Analytic plan
Multiple logistic regression models were used to assess the association between 1) PREDICT score and mortality within 6 months, and 2) PREDICT score and mortality within 12 months after index discharge. The models controlled for other patient-level variables including age, gender, marital status, ethnicity, insurance status, and comorbidities including heart-related problems, diabetes, renal, cancer, paraplegia, hemiplegia, liver disease, and metastatic carcinoma. We also performed a sensitivity analysis by comparing the area under the Receiver Operating Characteristic curve (ROC), between PREDICT score and PREDICT category with the APR-DRG Mortality Risk Index. Finally, multiple logistic regression models were performed to assess the association between PREDICT score and rehospitalizations, after controlling for the above mentioned covariates. Rehospitalization was dichotomously defined as rehospitalizations in the hospital network in the 1 month and 6 months after index discharge (no/yes).

RESULTS
Of the 3,488 hospitalizations that met inclusion criteria, 97 did not have a PREDICT score and/or category and thus were excluded from analyses related to the PRE-DICT score/PREDICT category. Of the final sample, 2,657 (78.35%) patient visits were identified as low risk category, 372 (10.97%) were medium risk, and 362 (10.68%) were high risk. Median age of the low, medium, and high risk groups were 59, 72, and 76 years, respectively (see Table 1).

PREDICT tool and mortality
A total of 228 (7%) patients were deceased at 6 months and 299 (9%) patients were deceased within 12 months. Of the patients identified as high risk, 73 (20%) died within 6 months and 91 (25%) within 12 months, compared to 52 (14%) and 60 (16%) for the medium risk group, and 106 (4%) and 133 (5%) for the low risk group. Logistic regressions revealed that individuals had greater odds of dying within 6 and 12 months if they had a higher PREDICT score, advanced age, multiple comorbidities including heart related problems and liver disease, and evidence of a palliative care consultation (see Table 2). Results revealed that each numeric increase in PREDICT score was associated with a 1.06 increase in odds of mortality within 6 months (p < .001) and 12 months (p < .001).

Concurrent and predictive validity
All three risk predictors (i.e., PREDICT category, PREDICT score, APR-DRG Mortality Risk Index) predicted mortality at 12 months with ROC association rates significant at p < .001. We observed significant differences in ROC association rates between PREDICT category and APR-DRG Mortality Risk Index, and PREDICT score and APR-DRG Mortality Risk Index (see Figure 1). Results indicated that the PREDICT score (ROC association rate 0.767) performed slightly better than the existing APR-DRG Mortality Risk Index (ROC association rate 0.73; p = .026), while the PREDICT category (ROC association rate 0.69) performed slightly worse than the APR-DRG Mortality Risk (p = .045).
Overall, there was slightly greater predictive ability of mortality within 12 months by using PREDICT score compared to the APR-DRG Mortality Risk.

PREDICT tool and rehospitalizations
We also examined the relationship between the PREDICT tool and rehospitalizations (see Table 3). Of the high risk group, 25% were rehospitalized within 1 month and 45% within 6 months. Two logistic regression models tested the effect of total PREDICT score on rehospitalizations within 1 month and within 6 months, while controlling for demographic and clinical variables. PREDICT score significantly predicted rehospitalizations; specifically, each unit increase in PREDICT score was associated with a 1.06 increase in odds of rehospitalization within 1 month (p < .001) and 1.05 increase in odds of rehospitalization within 6 months (p < .001).

Validation of the PREDICT tool
The PREDICT tool in an automated form predicts risk of mortality in patients admitted to a network of acute care hospitals. Our study is the first to include PREDICT as a tool embedded in an EMR across 8 hospitals. We sought to validate the PREDICT tool by assessing its predictive ability with mortality within 6 and 12 months, comparing it against an existing mortality risk index, and examining its utility in predicting rehospitalizations.
Findings indicated that the PREDICT score significantly predicted mortality within 6 and 12 months, after controlling for salient clinical variables. Of clinical relevance, the odds of mortality increased 1.06 times for every unit increase in total PREDICT score, meaning a patient with a total score of 15 would be 12.72 times more likely to be deceased within 6 and 12 months compared to a patient with a total score of 3. The ability to accurately identify patients at greater risk for mortality is important given that the PREDICT algorithm was implemented into the EMR EPIC system and can yield real-time scores to inform providers without additional manual data entry by clinicians. As previously mentioned, other existing mortality risk tools rely on clinician assessment, data entry, claims data unavailable at the time of admission, or are specific to populations and settings. The PREDICT tool, on the other hand, utilizes already available patient data and recalculates total scores in real-time, informing the clinician of any changes as they occur.
When compared to an existing mortality index, the PRE-DICT tool performed superior to or equal to the predictive index of the APR-DRG-Mortality Risk Index in predicting death at 12 months. Study findings are comparable to other validated 1-year mortality risk tools for older adults (C statistics range from 0.64 -0.79) [4][5][6][7][8] but less than the original PREDICT study from Australia (current 0.76 vs. original 0.86). [12] The difference may be due to the earlier utilization of palliative care services within our network of hospitals compared to utilization of palliative care in Australia. Our results also differed from prior studies as our sample had a lower death rate compared to the original PREDICT study (9% vs. 23%, respectively). [12] This is likely because we included all adult patients hospitalized while prior studies restricted study population to patients aged 55 and older. [12][13][14] Despite this sample expansion, our study is the first to validate an automated version of PREDICT within the EMR across a network of hospitals within the United States.

PREDICT and rehospitalizations
The current study is novel in demonstrating the predictive validity of PREDICT for rehospitalization. The PREDICT score was a significant predictor of rehospitalization both within 1 month and 6 months. Utilizing an automated risk tool for readmissions and mortality may allow researchers to identify high risk individuals at time of hospital admission and prospectively evaluate care, provide counseling, and administer disposition interventions during the index hospitalization which, in turn, may prevent readmissions. Designing real-time notifications to admitting physicians for the high risk group, has the potential to improve goals of care conversations, advance care planning, as well as utilization of palliative care and hospice that may in turn reduce burdensome hospitalizations at the end of life. [14][15][16][17]

Limitations
Our study is not without limitations. Despite the use of multiple hospitals, the hospitals are within one hospital network.
It is possible that patients had hospitalizations or medical information at non-network sites underestimating rehospitalization. Additionally, further calibration of the tool may be needed to better define the PREDICT variables within the EMR. For example, work is being done to refine the cancer variable to exclude cancer in situ, non-melanoma skin cancers, and prior resected cancers that were included in the PREDICT cancer variable. This modification will likely further improve tool performance and warrants ongoing scrutiny.

CONCLUSIONS
Our findings demonstrate progress towards developing and implementing tools within the EMR to aid clinicians in prognostication. With appropriate calibration, the EMR-driven PREDICT tool has the potential to be implemented and scaled across diverse clinical sites. It has advantages over other tools as it can be easily applied at time of hospital admission to a broad range of patients, and provides clini-cians real-time decision support to prompt earlier goals of care conversations, and earlier access to palliative care and hospice services. Further study is warranted to validate the PREDICT tool in other settings and evaluate its impact on clinical decision-making.

CONFLICTS OF INTEREST DISCLOSURE
The authors declare they have no conflicts of interest.