{'fraction_non_alphanumeric': 0.032052934605138785, 'fraction_numerical': 0.013921113689095127, 'mean_word_length': 4.971267316572601, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Early triage of patients diagnosed with COVID-19 based on predicted prognosis: A Korean national cohort study

We developed a tool for early triage of a COVID-19 patient based on a predicted prognosis, using a Korean national cohort of 5,596 patients. Predictors chosen for our model were older age, male sex, subjective fever, dyspnea, altered consciousness, temperature ≥ 37.5°C, heart rate ≥ 100 bpm, systolic blood pressure ≥ 160 mmHg, diabetes mellitus, heart disease, chronic kidney disease, cancer, dementia, anemia, leukocytosis, lymphocytopenia, and thrombocytopenia. Our model was better in predicting prognosis than protocols that are not based on data. The AUC of our model utilizing all the selected predictors was 0.907 in predicting whether a patient will require at least oxygen therapy and 0.927 in predicting whether a patient will need critical care or die from COVID-19. Even with age, sex, and symptoms alone used as predictors, AUCs were ≥ 0.88. In contrast, the protocols currently recommended in Korea showed AUCs less than 0.75.


Introduction
Since the World Health Organization (WHO) declared the coronavirus disease 2019 (COVID- 19) a pandemic in March 2020, it has been raging on, taking the lives of many people (over 1.32 million as of November 17, 2020) 1 . However, since no effective anti-viral drug or vaccine has been developed yet, the treatment mainly relies on symptomatic relief and supportive care, oxygen therapy, and critical care, depending on the disease severity. Thus, it is crucial to triage COVID-19 patients rapidly and efficiently so that limited medical resources, including quarantine facilities, hospital beds, and critical care equipment, can be allocated appropriately.
The current protocols recommended for triage and referral of COVID-19 patients in many countries or by WHO are based on known risk factors and expert opinion but have not been validated on the actual patient data [2][3][4][5] . Furthermore, since sudden disease progression in initially mild or asymptomatic COVID-19 patients is not rare with reported incidences of 6 to 12% 6-9 , we should base the triage and referral of COVID-19 patients on the worst severity expected during the disease course, rather than the severity at the time of diagnosis.
The data accumulated for several months now have enabled development of such a data-driven prediction model. Several prediction models for disease severity in COVID-19

Variables in four different tiers based on accessibility
We intended to develop a model that can be used flexibly in real-world circumstances where some of the variables may not be available. Therefore, we categorized variables into four tiers based on their accessibility (Table 1 and Fig. 1).

Tier 1: Basic demographics and symptoms
Tier 1 variables can be obtained by simply asking a patient questions: age, sex, body mass index (BMI), pregnancy, and symptoms. The symptoms included were subjective fever, cough, sputum, dyspnea, altered consciousness, headache, rhinorrhea, myalgia, sore throat, fatigue, nausea or vomiting, and diarrhea. We separated this group of variables from others because there could be times when we need to triage a patient quickly without physical contact.

Tier 2: Underlying diseases
Tier 2 variables are underlying medical conditions: hypertension, diabetes mellitus (DM), heart disease, asthma, chronic obstructive pulmonary disease (COPD), chronic kidney disease (CKD), chronic liver disease, cancer, autoimmune disease, and dementia. We categorized these variables into a separate group because sometimes patients may not know exactly their underlying medical conditions. In this case, further actions may be required, including reviewing medical records or other examinations.

Tier 3: Vital signs
Tier 3 variables are blood pressure, body temperature, and heart rate. Our data lacked information on breathing rate. We separated these variables from the first two tiers because these can be obtained only when a patient visits a medical facility or can measure their vital signs on their own. Blood pressure and heart rate were transformed into binary categorical variables by merging categories that were not significantly associated with disease severity based on the preliminary results in the training cohort: severe hypertension (systolic blood pressure ≥160 mmHg) and tachycardia (heart rate ≥100 bpm). We assumed that many patients had their body temperature measured while taking antipyretics, although our data did not contain the information on such patients' proportion.

Predictor selection
To identify robust and stable predictors, we repeated 10-fold cross-validation (CV) 100 times with shuffling and choose variables that were selected more than 900 times out of 1,000 trials (>90%) based on two algorithms: Least Absolute Selection and Shrinkage Operator (LASSO) and Random Forest (RF). A variable was selected if its coefficient was non-zero on LASSO, and its variable importance on RF was positive 16,17 .

Development of prediction models
We used four machine learning algorithms: ordinal logistic regression (OLR), multivariate RF, linear support vector machine (L-SVM), and SVM with the radial basis function kernel (R-SVM). For each algorithm, five models were created using one of the following five predictor sets: predictors chosen from the Tier 1 variables (Model 1), Tiers 1/2 variables (Model 2A), Tiers 1/3 variables (Model 2B), Tiers 1/2/3 variables (Model 3), and Tiers 1/2/3/4 variables (Model 4). We optimized the hyperparameters for RF and SVM through a 10-fold CV with a grid search in the training cohort, using the area under the receiver operator characteristics curve (AUC) as an evaluation metric.

Validation of prediction models in comparison with current protocols
We validated the optimized models in the test cohort after fitting them onto the entire training dataset. Based on the probabilities for each outcome category, we assessed the diagnostic performance of each model for whether or not a patient will require treatment (Outcome 1 vs. 2/3), and whether or not a patient will require critical care or die (Outcome 1/2 vs. 3). Sensitivity, specificity, accuracy, precision, and negative predictive value (NPV) according to different probability cutoffs were calculated, in addition to AUC. We also drew calibration curves to compare the predicted and observed probabilities visually. device, it can be coded to choose an appropriate model automatically depending on available predictors; a simplified Python code with coefficients trained onto the entire dataset can be found in *** blinded ***.

Discussion
Our results demonstrate that a data-driven model to predict prognosis can be a good tool for early triage of COVID-19 patients. A significant shortcoming of the triage protocols that are not based on data is that risk factors are not weighted appropriately based on their effects on the outcome. For example, the WHO algorithm for COVID-19 triage and referral regards age > 60 years and the presence of relevant symptoms or co-morbidities as risk factors, but it does not put different weights on them 2 . However, if not treated as a continuous variable, age should be divided into multiple categories with appropriate weights because the risk continues to increase with age even after 60 years. Different symptoms or co-morbidities must also be weighted according to their importance when assessing the patients' status for triage. For example, in the current study, subjective fever, dyspnea, and altered consciousness were independent risk factors for severe illness, while other symptoms such as cough, sputum production, sore throat, myalgia, and diarrhea were not. Our final prediction model used the OLR algorithm. We chose the OLR over the other machine learning algorithms (i.e., RF, L-SVM, and R-SVM) because it showed comparable or superior performances to the other algorithms in the final evaluation. Furthermore, a linear model like the OLR is more interpretable and easier to use even without a computer device, as nomograms can be used instead. We also observed the linear model's superiority in predicting COVID-19 prognosis in our previous study in which we developed a model to predict the risk of COVID-19 mortality based on demographics and medical claim data 15 .
A difference of the current model from other earlier models is that we divided disease severity into three categories. This is more helpful than the binary categorization (i.e., recovery vs. mortality), because not all medical facilities capable of oxygen therapy can also provide critical care, such as mechanical ventilation or ECMO. Furthermore, our model uses different algorithms depending on the available variable subsets. Health workers sometimes need to triage newly diagnosed COVID-19 patients even by a phone call alone in the real-world field, and patients commonly do not know their underlying disease exactly. Therefore, we expect that our model's flexibility may lead to a more widespread use.
The predictors chosen in this study are not much different from the known risk factors of developing into critical conditions from COVID-19 19 . However, it was unexpected that COPD, a known strong risk factor, was not selected as a predictor. We assume that this is because there were only 40 patients with COPD in the entire cohort, of whom 65% had dyspnea, and the disease severity of COPD might have varied widely. Thus, it is likely that the number of COPD cases was too small (even smaller in the training cohort after the training-test set split) to play a significant role independently from the other strong predictors. We hope to have more confirmatory results through further investigation as the KDCA plans to release the enhanced data with more patients soon.
There are limitations to our current model. First, we need to develop a more robust model by enrolling more patients and conduct prospective validation. We plan to use the current model in actual practice and keep improving the model using the newly accumulated data.
Second, since we trained our model on Koreans' data, it is unsure whether it can be generalizable to patient cohorts in other countries or races. We hope to be able to develop a triage model that can be used globally through collaboration. Lastly, our data lacked some important variables, such as smoking, respiratory rate, and oxygen saturation, and had missing values in some of the Tiers-2/3/4 variables, which may have affected the training and performance of the algorithms using those variables. We did not perform imputation for missing values because we did not want the uncertainty and potential bias from imputation, and imputation for missing values did not make significant differences in our preliminary analysis.
In conclusion, we developed a set of models that can be used for disease severity prediction and triage or referral of COVID-19 patients. Our prediction model has a good performance even with age, sex, and symptoms alone. The model performance can be enhanced if further information on underlying disease, vital signs, or blood test results is available. Acknowledgement *** blinded *** The authors would like to thank the KCDA for the support and data they kindly provided. The authors alone are responsible for the content of this article.

Author contributions
*** blinded ***   The results of other machine learning algorithms can be found in Supplementary Table 5     shows that a patient who is older than 50 years and has relevant symptoms or underlying diseases is more likely to require oxygen therapy or critical care.