{'fraction_non_alphanumeric': 0.0228521839745444, 'fraction_numerical': 0.0145357246167197, 'mean_word_length': 5.073342116820378, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction of emergency department resource requirements during triage: An application of current natural language processing techniques

Abstract Objective Accurate triage in the emergency department (ED) is critical for medical safety and operational efficiency. We aimed to predict the number of future required ED resources, as defined by the Emergency Severity Index (ESI) triage protocol, using natural language processing of nursing triage notes. Methods We constructed a retrospective cohort of all 265,572 consecutive ED encounters from 2015 to 2016 from 3 separate clinically heterogeneous academically affiliated EDs. We excluded encounters missing relevant information, leaving 226,317 encounters. We calculated the number of resources used by patients in the ED retrospectively and based outcome categories on criteria defined in the ESI algorithm: 0 (30,604 encounters), 1 (49,315 encounters), and 2 or more (146,398 encounters). A neural network model was trained on a training subset to predict the number of resources using triage notes and clinical variables at triage. Model performance was evaluated using the test subset and was compared with human ratings. Results Overall model accuracy and macro F1 score for number of resources were 66.5% and 0.601, respectively. The model had similar macro F1 (0.589 vs 0.592) and overall accuracy (65.9% vs 69.0%) compared to human raters. Model predictions had slightly higher F1 scores and accuracy for 0 resources and were less accurate for 2 or more resources. Conclusions Machine learning of nursing triage notes, combined with clinical data available at ED presentation, can be used to predict the number of required future ED resources. These findings suggest that machine learning may be a valuable adjunct tool in the initial triage of ED patients.

resources. These findings suggest that machine learning may be a valuable adjunct tool in the initial triage of ED patients.

K E Y W O R D S
emergency department, machine learning, resources, triage, natural language processing

BACKGROUND
The number of emergency department (ED) visits in the United States is continuing to rise. Given this influx of patients into the ED, crowding and overall ED efficiency have become focal points of discussion. Indeed, crowding in the ED has been associated with negative outcomes, including increased morbidity and mortality, longer wait times, increased length of stay, increased hospital expenses, and poorer patient perceptions of care. 1-4 Accordingly, a substantial emphasis has been placed on optimizing ED efficiency by appropriately allocating departmental resources and patient distributions. To predict and potentially prevent downstream bottlenecks in the ED, past studies have demonstrated that it is possible to use the information available at the time of patient triage to make predictions regarding the severity and final ED disposition of patients. [5][6][7][8][9] Triage documentation is typically written by a nurse who interviews and examines the patient briefly in a small examination area adjacent to the ED waiting room to determine severity and likely number of resources required. The nurse records various clinical data points (eg, vital signs, weight, comorbidities). The nurse then typically documents a short 1 to 3 sentence triage note summarizing the patient's reason for presentation and then assigns a triage acuity level from 1 to 5 based on the Emergency Severity Index (ESI) algorithm. 10 The ESI is used to stratify patients into triage groups from 1 (most urgent) to 5 (least urgent) based on acuity of illness and resource needs and has been validated multiple times in a variety of clinical settings. It is the most common triage algorithm in the United States. 10 Resources are broadly grouped into 0, 1, or 2 or more. Examples of an ED resource, according to ESI, include a laboratory test (blood or urine) or a laceration repair.
Some typical ED activities, such as providing oral medications or placing a splint, are not considered resources. Without training, this list is not necessarily intuitive.

Importance
Nursing triage documentation is the first available medical assessment in the electronic medical record and is typically written by experienced ED nurses and may contain higher level information (ie, sentiment) regarding the patient's condition. We recently demonstrated that natural language processing (NLP) coupled with machine learning using nursing triage notes can be used to predict discrete outcomes such as hospital admission. 11 Machine learning excels at predicting quantifiable metrics. An advantage of using information at the time of triage is that it is present in the electronic medical record hours before physician documentation is available.

Goals of this investigation
We hypothesized that NLP coupled with deep learning of triage notes and other clinical data would be able to produce a model that is predictive of the number of resources required for a given patient in the ED.
To our knowledge, the current study is the first to use NLP and machine learning of clinical data to predict the number of resources required for patients in the ED.

Study design and setting
A retrospective cohort was constructed using electronic health record data from all 260,842 consecutive ED encounters from 3 separate clinically heterogeneous academically affiliated EDs from 2015 to 2016.
Hospital A is a community hospital in an urban setting with a patient volume of ≈75,000 ED visits per year. Hospital B is a community hospital in a suburban setting with a volume of ≈27,000 visits per year.
Hospital C is a quaternary care academic medical setting in a major metropolitan area with an ED patient volume of ≈40,000 visits per year. The research study protocol and procedures were reviewed and approved by the institutional review board.

Selection of participants
We excluded encounters based on frequent use (defined as ≥20 visits during the study years, to minimize training bias), disposition other than admission or discharge, and arrival time unavailable. The final cohort of interest included 226,317 unique clinical encounters from 144,421 unique patients.

Interventions
The study was retrospective in nature. Accordingly, no interventions were performed on the study subjects.

Measurements
For this analysis, we included the nursing triage notes in word token format and clinical variables that were available at the time of triage, including demographics, age, sex, vital signs, number of ED visits within the past 12 months, number of ED visits within past 7 days, pain score, disposition from most recent encounter, arrival mode, whether the patient had altered mental status, one-hot encoded chief complaint, past medical problems, most recent medical problem, and home medications.

Outcomes
The outcome of interest was model performance, which was measured by accuracy (percent correct predictions over all predictions) and F1 score.

Analysis
To use free-text natural language data as input for training and testing the model, it was first necessary to convert each triage note to a fixed-length vector to serve as numerical representations of the free text in the notes. Briefly, the pretrained word embeddings were used to transform the free text from the triage notes into word vectors of 200 length. First, common stop words were removed. Second, each word was tokenized to correspond to a row in the pretrained embedding matrix. As opposed to randomly initializing word vectors, we used pretrained embeddings published by, 12 which were generated from PubMed abstracts and a large text corpus from Wikipedia using the word2vec program to facilitate the learning process. 13 Long short-term memory models have been used extensively in deep learning of natural language data and are variants of recurrent neural networks that use memory gates to update the model's internal representation using a single input word at a time. 14 Tokenized triage notes were used as an input to an embedding layer with weights from the embedding matrix in the aforementioned step. This embedding layer was subsequently used as the input to a long short-term memory layer with output dimensionality of 128. This long short-term memory output was concatenated with 9,650 clinical variables that were described previously.
We defined 3 possible categories for number of resources: 0 To provide a real-world comparison of predictive performance, 2 experienced ED nurses were blinded to the number of resources and were tasked with predicting number of resources (by categories of 0, 1, or 2 or more). A total of 1,000 encounters were randomly selected without replacement from the test data set and were rated by both nurses. The nurses were given typically available clinical information at the time of triage (Table 1) as well as comorbidities, the ED nursing triage note, past medical history, whether the patient had altered mental status, ED visits within the past 7 days, ED visits within the past 12 months, arrival mode, and most recent problem. These were provided in an attempt to approximate the data they would have at hand in an ideal scenario to assist with triage. Model predictions were obtained for each of these encounters and were compared against human rater predictions. Interrater reliability for human raters was calculated using the κ statistic.

Main results
Metrics were computed to gauge overall model performance, and a confusion matrix was constructed to determine accurate classification versus misclassification of model predictions (  Figure 2B).
The accuracies of the first and second raters for the prediction of number of resources category were 67.8% and 70.1%, respectively.
The combined accuracy of both raters was 69.0%, and agreement (κ statistic) was 0.588 ( Figure 3)

Limitations
The current study had several limitations. First, the design of this study was retrospective in nature. Although we excluded encounters that resulted in dispositions other than admission or discharge, it is TA B L E 2 F1 scores across each "number of resources" category based on predictions of the trained model using all test set data, the trained model using all validation data set encounters, and the trained model and human raters using 1,000 encounters randomly selected from the test set

DISCUSSION
In this study, we used triage notes from 226,317 patient encounters across 3 large metropolitan EDs to predict the number of resources (0, 1, or 2 or more) required for patients during their ED visits. A particular advantage of using triage notes, vital signs, and known clinical history is that this information is typically available at the time of triage. Accordingly, these data are able to be used early in the encounter to make predictions about likely events during the ED visit. In the current study, we demonstrated that the free text from nursing triage notes can be converted into numerical representations of natural language and may then be combined with other quantitative and categorical clinical data to predict the number of resources required by a given patient during an ED visit. These findings suggest that free-text data contained in patient charts, such as the triage notes used in this study, carry information that should be considered and may be useful when developing predictive models of an anticipated course in the ED. convert free text to quantitative data in a formulaic and repeatable process.
The context of NLP and modeltraining may influence the final numerical representations of natural language. For example, triage notes contain specific language and symbols that are relatively uncommon in everyday English language. In addition, many words and phrases that are used in triage notes carry different meanings in the healthcare setting (ie, "complains of," "endorses," and "stable"). The current study used word embeddings generated from a large collection of PubMed abstracts and Wikipedia articles to capture language in medical and general contexts. 9 Although the current study is limited to the use of machine learning and NLP for prediction at an individual patient level, it is worth noting that these technologies may have particular relevance to the operation of health systems. For example, modern health system design may incorporate concepts from resilience engineering, 18 wherein systems are designed to be able to proactively adjust to anticipate a stressor, absorb the stressor, and return to a baseline state of functioning.
Machinelearning tools, in such systems, might be used to anticipate impending stressors and assist in the allocation of resources.
In this study, we used machine learning to predict the number of resources needed for patients during their ED visits via NLP of nursing triage notes in combination with other current and past clinical data.
The results suggest that the number of resources is predictable via machine learning using these data, which are typically available at the time of triage. This has important clinical implications, as machine leaning may hold potential to augment triage capabilities for triage staff, inform allocation of ED clinical and staff resources, and improve flow in the ED. External prospective studies will be required to validate these findings and further explore the role of machine learning in the ED triage process.

CONFLICT OF INTEREST
J.S. is the cofounder of Vital Software, Inc, a company engaged in developing artificial intelligence clinical decision support products for the emergency department. The other authors declare no conflicts of interest.

AUTHOR CONTRIBUTIONS
NS, FB, RP, and JS were involved in designing the paper. NS, FB, MD, MK, MB, and JS helped in the execution of the work. All the authors contributed in writing the paper.