{'fraction_non_alphanumeric': 0.023758218686115256, 'fraction_numerical': 0.014089176197579977, 'mean_word_length': 5.273830155979203, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep‐learning approaches to identify critically Ill patients at emergency department triage using limited information

Abstract Study objective Triage quickly identifies critically ill patients, facilitating timely interventions. Many emergency departments (EDs) use emergency severity index (ESI) or abnormal vital sign triggers to guide triage. However, both use fixed thresholds, and false activations are costly. Prior approaches using machinelearning have relied on information that is often unavailable during the triage process. We examined whether deep‐learning approaches could identify critically ill patients only using data immediately available at triage. Methods We conducted a retrospective, cross‐sectional study at an urban tertiary care center, from January 1, 2012–January 1, 2020. De‐identified triage information included structured (age, sex, initial vital signs) and textual (chief complaint) data, with critical illness (mortality or ICU admission within 24 hours) as the outcome. Four progressively complex deep‐learning models were trained and applied to triage information from all patients. We compared the accuracy of the models against ESI as the standard diagnostic test, using area under the receiver‐operator curve (AUC). Results A total of 445,925 patients were included, with 60,901 (13.7%) critically ill. Vital sign thresholds identified critically ill patients with AUC 0.521 (95% confidence interval [CI] = 0.519–0.522), and ESI <3 demonstrated AUC 0.672 (95% CI = 0.671–0.674), logistic regression classified patients with AUC 0.803 (95% CI = 0.802–0.804), 2‐layer neural network with structured data with AUC 0.811 (95% CI = 0.807–0.815), gradient tree boosting with AUC 0.820 (95% CI = 0.818–0.821), and the neural network model with textual data with AUC 0.851 (95% CI = 0.849–0.852). All successive increases in AUC were statistically significant. Conclusion Deep‐learning techniques represent a promising method of augmenting triage, even with limited information. Further research is needed to determine if improved predictions yield clinical and operational benefits.

network model with textual data with AUC 0.851 (95% CI = 0.849-0.852). All successive increases in AUC were statistically significant.

Importance
Both ESI and vital sign triggers rely on specific vital sign thresholds.
Although ESI is among the most validated algorithms for triage, numerous studies have shown that both under-triage and over-triage remain persistent issues. 3,4 When patients are under-triaged, opportunities to change the course of disease are missed, whereas over-triage has the potential to disrupt physicians' and nurses' workflows, detracting from safe and efficient care for other patients in the ED. In particular, better understanding of the effects of advanced age, the influence of specific chief complaints, and more robust criteria for vital sign abnormalities have been highlighted as areas for improvement to the current ESI. 4,5 Studies examining machine-learning approaches have shown promise in supplementing the ESI score at triage, including random forest models to help differentiate outcomes for patients within ESI categories, 6 gradient boosting algorithms to predict admission, 7 and outcomes in specific conditions, such as mortality in sepsis. 8 However, many of these studies have leveraged information, such as structured diagnosis lists and past medical histories, which is unavailable for many patients at the time of triage. In particular, patients who are making their first contact with an ED rarely bring medical history information in a readily accessible electronic format. Patients also may be unable to meaningfully provide a clear past medical history, whether because of dementia, language barriers, or limited health literacy.
Depending on their complexity, some machine-learning approaches are not readily integrated with commercial electronic health record (EHR) systems, and may require considerable effort to tune and adapt to a health system's specific population.

Goals of this investigation
We examined whether a set of progressively complex deep-learning algorithms could identify critically ill patients with greater discriminative power than ESI or vital sign triggers alone using information immediately available at triage, as measured by the area under the receiveroperator curve (AUC).

Study design and selection of participants
This was an observational study examining a retrospective cohort of adult patients who visited an academic, urban ED at a tertiary care center in the Northeastern United States with an average volume of 55,000 visits annually. All patients between January 1, 2012-January 1, 2020 were screened for the study. Patients were included if their data included triage vital signs (patients with one or more vital signs were included, patients with none were not). ESI score, whether a vital sign trigger had been activated, and ultimate disposition (including whether they expired within the ED) were recorded for all patients.
Clinical data was obtained from an automated quality assurance database for the ED information system. Data were de-identified during extraction using the HIPAA SAFE HARBOR method, and the study authors were blinded to identifying information. The study was structured with respect to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) statement. 9 The host institutional review board granted an exemption for de-identified data used.

Measurements
Vital signs were defined as a patient's initial triage measurement of temperature, heart rate, systolic and diastolic blood pressure, respiratory rate, and percent oxygenation. To exclude potentially erroneous data entries, vital signs were included in the analysis provided they fell within broad physiologically feasible ranges, including temperature below 110 • , heart rate below 300 beats/min, systolic, and diastolic measurements below 300 mm Hg, respiratory rate below 80 respirations/min, and oxygenation at or below 100%.
Missing or spurious data were considered as null values for the analysis.
Triage chief complaints were included as free text entered immediately at a patient's arrival at triage. Chief complaints were not standardized during the study period, and typographic errors and blank entries were included in the analysis for fidelity. Nursing documentation after the patient's initial registration was not included. As a result of de-identification, patients older than 89 years of age were included within a single 90+ age category.
Vital sign trigger criteria were defined as heart rate <40 or >130 beats/min, respiratory rate below 8 or above 30 respirations/min, systolic blood pressure below 90 mm Hg, or an oxygen saturation below 90% on room air, which is standard practice for trigger activations at triage for the institution. 2 As a result of the clinical system at our institution, all patients who meet clinical trigger criteria are classified as having an ESI score of 1 or 2.

Outcomes
The primary outcome was whether a patient was critically ill, defined as whether they expired within 24 hours of arrival, required ICU admission from the ED, or were transferred from an inpatient ward to the ICU (or for an emergent procedure) within 24 hours of admission. The data abstraction process included the possibility of a patient being discharged and returning to the ED as critically ill within 24 hours. There have been many different measures of resource use and illness severity used across studies evaluating the efficacy of triage and machine-learning predictions, including specific diagnoses such as sepsis, 8 admission, 7 and overall mortality. 10 The composite of mortality and ICU admission is an appealing compromise metric, as it identifies a population that is more likely to require rapid intervention than the larger population of patients needing admission.

The Bottom Line
Initial patient triage is designed to rapidly identify which patients will require the greatest resources in the emergency department and the hospital. In this manuscript, the authors demonstrate that deep learning techniques can improve the accuracy of model to predict the need for ICU admission and or death when compared to traditional triage methods.
Future prospective studies are needed to determine the benefit of these new models.

Neural network model creation and derivation
Neural networks consist of a series of nodes known as neurons, which take input variables, apply an affine transformation to the inputs based on a set of weights, and yield an output based on whether a discrete threshold, known as the activation function, has been met. A loss function measuring the neuron's output relative to the correct diagnostic labels is used to adjust the neuron's weights repeatedly until loss has been minimized. Accordingly, logistic regression can be thought of as a single neuron with a sigmoid activation function.
Neural networks leverage multiple layers of interconnected neurons to model high-degree interaction effects. For instance, a neuron within a deep neural network layer might recognize specific combinations of elevated heart rate and temperature, whereas a separate neuron within the same layer might recognize separate thresholds for the same variables when associated with a different age range or set of chief complaints.
Our models were created in TensorFlow, an open-source framework for deep-learning. 11 The triage data was split between the structured vital sign data and the chief complaint text data. The vital sign data was normalized and used as the input for a logistic regression (with L2 normalization), and for a 2-layer deep neural network. A third, deep neural network combining both structured and text data likewise used the vital sign data as the input to a smaller 2-layer deep neural network. The chief complaint text data was first embedded into a text vector and input into a long short-term memory network, a standard architecture for text processing and in clinical natural language processing. 12,13 As tree-based approaches have been used in a number of other recent studies of machine learning in the ED, 6 we also provided a tree-based model on the structured data for comparison, using the XGBoost framework. 14 The outputs of these 2 subnetworks were then concatenated and used as the cumulative input to a densely connected layer, with a final logistic prediction layer. Hyperparameters of the model, such as the number of neurons per layer and the total number of layers in the overall model, were adjusted using random search and subsequently manually adjusted, with adaptive moment estimation for optimization. 15 Data was divided into a randomized 80:10:10

Analysis
We report the diagnostic accuracy of the methods evaluated (both current triage methods and machine-learning methods) in this study in terms of their sensitivity, specificity, accuracy, and AUC with 95% CI, relative to the reference standard of critical illness abstracted from the medical record. Statistical analysis was carried out in Python 3.8 using the SciPy and SciKit-Learn scientific and machine-learning libraries. 16,17 Differences between group means were tested at an alpha level of 0.05, with strict Bonferroni correction for multiple com-

Characteristics of the study subjects
From January 1, 2012-January 1, 2020, 445,925 adult patients met inclusion criteria, detailed in Figure 1A. Patient demographic and vital sign characteristics are detailed in Table 1 were included in the analysis. The full enrollment process is described in Figure 1.
There were significant differences between the groups of patients assessed at triage who were critically ill and those who were not. Critically ill patients typically were older, more likely to be male, had faster heart rates and respiratory rates, higher temperatures, lower blood pressures, and lower oxygen saturations, all of which were significant

Main results
The existing standards of triage evaluation to identify critical patients at our institution, abnormal vital sign triggers and ESI scores ≤2,  Table 2 and ROC curves in Figure 2.

DISCUSSION
Our study examines several models of triage that demonstrate increasing accuracy in tandem with increasing complexity. For those models relying on vital sign data alone, this progression is logical, as more complex models can draw more granular borders between data. The discrete vital sign cutoffs used for trigger criteria are simple to remember and demonstrate considerable specificity but miss a substantial number of critically ill patients. The enhanced accuracy of the more complex models likely reflects 2 factors-interaction effects between different vital signs and the potential effects of age.
Early in their disease course, a critically ill patient may demonstrate subtle changes in multiple vital signs, which may be difficult to recognize individually, but meaningful in the aggregate. We are primed to recognize that a heart rate above 100 or a respiratory rate above 20 is abnormal because these numbers are salient. 21 The modest improvement in accuracy between the logistic regression model and the neural network model and tree-based models examining vital signs and demographic variables alone may reflect an underlying information-theoretic limit to a single set of measurements.
Although these models are more accurate than rigid vital sign cutoffs and use of the ESI score, it is notable that optimal neural network model examining vital signs in our study was only 2 layers deep and was not improved by adding additional layers representing higher-dimension interaction effects. This suggests that although vital signs are essential to the triage process, and their interpretation can be improved substantially, alone they are not sufficient to identify all critical patients.
Although the addition of textual chief complaint data entails only a small amount of additional data per patient, it was associated with small but significant improvements in both the sensitivity and specificity of the neural network model. This likely reflects the critical contextual information that a patient's chief complaint provides. A young patient presenting with tachycardia and tachypnea may not be critically ill if their complaint is anxiety, and the additional attention of being taken to a critical care bay might exacerbate their symptoms.
However, the same vital signs in a patient with a chief complaint of abdominal pain could be essential to identifying a ruptured ectopic pregnancy. Integrating free-text data directly into the analysis is a particular strength of the neural networks, as many other approaches (eg, regression) may require either entering completely standardized chief complaints (for use as categorical variables), or extensive pre-processing. For example, a system that requires a nurse to choose between "chest pain" and "back pain" will not capture the information that "sudden chest pain radiating to back" can signify as a chief complaint.
Our models demonstrate similar levels of accuracy to prior machinelearning approaches to predict admission from the ED.

LIMITATIONS
This was a retrospective study conducted at a single urban tertiary care center with a significantly higher proportion of critically ill patients than has been reported in similar studies. This may be partly explained in terms of the relatively high average age of the patient population but could also reflect institutional bias. The use of ICU admission as a proxy for critical illness also introduces several significant limitations.
Depending on a hospital's ICU and floor capabilities, as well as its institutional culture, a patient requiring ICU admission at one institution An important technical and clinical consideration is that although our study examined tests in terms of AUC and optimal test characteristics, many clinicians will prefer to use diagnostic thresholds that maximize test sensitivity at triage at the expense of reduced specificity. These preferences, and their operational consequences, should be examined carefully before adapting any clinical decision support system.

CONCLUSIONS
In this single-center, retrospective study of deep-learning approaches to identifying critically ill patients at ED triage, neural network and gradient-boosting models demonstrated significantly higher accuracy than traditional methods of triage, suggesting that these models have the potential to significantly enhance the triage process. Although diagnosing patients who are critically ill can more accurately help to more appropriately mobilize resources within the ED to treat them, future studies are needed to assess the clinical and operational impact of using neural networks to enhance the triage process and to identify which critically ill patients can benefit most from rapid intervention.

AUTHORS CONTRIBUTIONS
JWJ, ELL, LDS, and NE conceived of the overall study design. Initial JWJ drafted the initial manuscript. LJJ provided initial manuscript review. All authors contributed significantly to its revision. LDS and MWD provided supervision of the study. JWJ takes responsibility for the integrity of the data and the accuracy of the data analysis.

Risk Management Foundation of the Harvard Medical Institutions
Incorporated, was not involved in the design and conduct of the study; collection, management, analysis, or interpretation of the data; or preparation and review of the manuscript. CRICO did not approve the manuscript and had no input in the decision to submit the manuscript for publication. No publication restrictions apply. The content is solely the responsibility of the authors and does not necessarily represent the official views of Harvard University and its affiliated academic healthcare centers.

DATA AVAILABILITY STATEMENT
Source code for the triage neural network models are available at: https://github.com/jwjoseph/NN_triage_predict. Code is free for use and adaptation under the MIT License.