{'fraction_non_alphanumeric': 0.03530904477070872, 'fraction_numerical': 0.020047127061808955, 'mean_word_length': 5.379740980573543, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Early Prediction Model for Critical Illness of Hospitalized COVID-19 Patients Based on Machine Learning Techniques

Motivation Patients with novel coronavirus disease 2019 (COVID-19) worsen into critical illness suddenly is a matter of great concern. Early identification and effective triaging of patients with a high risk of developing critical illness COVID-19 upon admission can aid in improving patient care, increasing the cure rate, and mitigating the burden on the medical care system. This study proposed and extended classical least absolute shrinkage and selection operator (LASSO) logistic regression to objectively identify clinical determination and risk factors for the early identification of patients at high risk of progression to critical illness at the time of hospital admission. Methods In this retrospective multicenter study, data of 1,929 patients with COVID-19 were assessed. The association between laboratory characteristics measured at admission and critical illness was screened with logistic regression. LASSO logistic regression was utilized to construct predictive models for estimating the risk that a patient with COVID-19 will develop a critical illness. Results The development cohort consisted of 1,363 patients with COVID-19 with 133 (9.7%) patients developing the critical illness. Univariate logistic regression analysis revealed 28 variables were prognosis factors for critical illness COVID-19 (p < 0.05). Elevated CK-MB, neutrophils, PCT, α-HBDH, D-dimer, LDH, glucose, PT, APTT, RDW (SD and CV), fibrinogen, and AST were predictors for the early identification of patients at high risk of progression to critical illness. Lymphopenia, a low rate of basophils, eosinophils, thrombopenia, red blood cell, hematocrit, hemoglobin concentration, blood platelet count, and decreased levels of K, Na, albumin, albumin to globulin ratio, and uric acid were clinical determinations associated with the development of critical illness at the time of hospital admission. The risk score accurately predicted critical illness in the development cohort [area under the curve (AUC) = 0.83, 95% CI: 0.78–0.86], also in the external validation cohort (n = 566, AUC = 0.84). Conclusion A risk prediction model based on laboratory findings of patients with COVID-19 was developed for the early identification of patients at high risk of progression to critical illness. This cohort study identified 28 indicators associated with critical illness of patients with COVID-19. The risk model might contribute to the treatment of critical illness disease as early as possible and allow for optimized use of medical resources.


INTRODUCTION
The coronavirus disease 2019 (COVID-19) pandemic is spreading worldwide. As a communicable disease, COVID-19 is caused by severe acute respiratory syndrome coronavirus 2. Until 14 February 2022, the WHO reported 412,044,520 COVID-19 confirmed cases globally, with an average mortality rate of 1.4%. The clinical spectrum of COVID-19 infection ranges from asymptomatic infection, and mild upper respiratory tract illness to critically ill cases (1). It has been reported that about 5% of patients with COVID-19 infection experience rapid deterioration from the onset of symptoms into critical illness (2) and with a mortality rate of 61.5% for critical ones within 28 days of hospital admission (3). Treatment of patients with critical illnesses constitutes great pressure on medical services, especially results in the lack of intensive care resources. Therefore, early identification and effective triaging of patients with a high risk of developing critical illness COVID-19 upon admission can aid in improving patient care, increasing the cure rate, and mitigating the burden on the medical care system.
The risk factors for critical illness are not well-revealed. Previous reports have identified that older age, organ dysfunction, neutrophilia, preexisting concurrent cardiovascular or cerebrovascular diseases, coagulopathy, amounts of CD3+CD8+ T cells, and elevated D-dimer levels are associated with the development of acute respiratory distress syndrome and increased mortality risk (1,(4)(5)(6)(7)(8)(9). A limited number of publications have identified chest radiographic abnormality, older age, hemoptysis, dyspnea, unconsciousness, number of comorbidities, cancer history, neutrophil-to-lymphocyte ratio (10), lactate dehydrogenase (LDH), and direct bilirubin are risk factors associated with the development of critical illness (11,12). Clinical scores for predicting which patients with COVID-19 will develop critical illness were developed with these above 10 factors (11,12), which show well-discrimination. In addition, an integrated model was developed with patient history, laboratory markers, and chest radiography at hospital admission to predict critical illness by Schalekamp et al. (13). However, in these models, some diagnoses of co-existing illness and symptoms were from patients' self-reports at admission, which might lead to recall bias.
Mathematical modeling with appropriate inputs can make predictions in the dynamics and control of the infectious disease. A series of mathematical models have been developed on the transmission dynamics and control of COVID-19 or SARS-CoV-2 virus in different countries (14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24), namely, Wuhan, Italy, and the USA. In this retrospective multicenter study, we proposed and extended classical least absolute shrinkage and selection operator (LASSO) logistic regression for the early identification of patients at high risk of progression to critical illness. We systematically analyzed the accessible laboratory findings of confirmed 1,929 patients with COVID-19 having clear prognostic information in 32 hospitals in Hubei and Hunan provinces of China and identified robust and meaningful factors associated with a critical illness. The laboratory findings were measured objectively. A risk prediction model was constructed according to LASSO logistic regression to help identify patients at the time of hospital admission who are at high risk of developing a critical illness. This model aims at distinguishing patients at imminent risk of critical illness, thereby optimizing the allocation of limited healthcare resources and potentially lowering the mortality rate.

Data Collection
This study has been proved by the Institute of Clinical Pharmacology, Central South University. For the urgent need to collect and analyze data on this emerging pathogen, the ethics committee of the Institute of Clinical Pharmacology, Central South University granted a waiver of written informed consent from study participants. Medical records of hospitalized patients with COVID-19 diagnosed in 31 hospitals in China (4 hospitals in Hubei Province and 27 hospitals in Hunan Province) were collected. All patients who were diagnosed with COVID-19 by positive high-throughput sequencing or realtime reverse-transcription PCR (RT-PCR) assay for nasal and pharyngeal swab specimens were screened, our study enrolled all adult inpatients (≥18 years old) who were hospitalized for COVID-19 and had an explicit outcome of critical illness. The data were cross-checked by experienced respiratory clinicians. All patients with data on clinical status at hospitalization (laboratory findings, critical illness, and discharge status) were included.

Clinical Outcome
The outcome of this study is a critical illness, which is defined as a composite of invasive ventilation, admission to the intensive care unit (ICU), or fatal of patients with COVID-19 (25). The follow-up time was calculated from the first day of hospitalization to the date of death or discharge, or the censored date (12th April 2020 for the development cohort and 11 June 2020 for the validation cohort).

Potential Predictive Variables
Demographic variables and laboratory findings of patients at hospital admission were collected as potential predictive variables. Demographic variables included age and gender. Laboratory findings were conducted as the first measurement within 2 days after at admission, laboratory indexes with complete measurements for more than 50% of the patients in the development cohort were collected: hematologic (hematokrit, basophils, eosinophils, lymphocytes, monocytes, neutrophils, mean corpuscular volume, hemoglobin concentration, coefficient of variation [CV] and SD of red blood cell volume distribution width [RDW], blood platelet count, thrombocytocrit, red blood cell, and white blood cells), biochemical [levels of glucose, K, Na, total Ca, Cl, total protein, lactate dehydrogenase (LDH), glutamic-pyruvic transaminase, creatine kinase, aspartate transaminase (AST), creatine kinase muscle-brain isoform (CK-MB), creatinine, ureophil, albumin, globulin, albumin to globulin ratio, and glomerular filtration rate

Statistical Analysis
Continuous and categorical variables were presented as mean, SD [interquartile range (IQR)], and n (%), respectively.
A total of 1,255 patients hospitalized with COVID-19 in the development cohort were included for variable selection. To access the association between the quantitative laboratory findings described above and the occurrence of critical illness, a univariate logistic regression analysis was conducted. Since the odds ratio (OR) is interpreted per unit change, to standardize ORs between variables with a different range, logistic regression analysis was applied to dichotomies data (1 = with the occurrence of critical illness and 0 = without the occurrence of critical illness) with quartiles of each of the 38 laboratory findings modeled as continuous (<25% quartile = 1; ≥25% and <50% quartile = 2; ≥50% quartile and <75% quartile = 3; and ≥ 75% quartile = 4). The associations between the occurrence of critical illness and age (≥55 vs. <55 years) were also evaluated.
The statistically significant 28 covariates (p < 0.05) in the univariate logistic analysis were selected as candidates for risk score development of critical illness. A total of 1,064 patients with at least 80% data completeness of the above 28 variables were utilized for model establishment. We applied predictive mean matching to impute numeric features (laboratory findings) with "mice" packages in R for these 1,064 patients.
Prediction models were developed with the LASSO logistic regression, support vector regression (SVR), artificial neural network (ANN), regression tree (RT), and multivariate adaptive regression splines (MARS) machine learning techniques. We used the "glmnet" (14) package for LASSO, "e1071" package for SVR, "RSNNS" package for ANN, "rpart" package for RT, and "earth" package for MARS. Default parameters were used. L1-penalized least absolute shrinkage and selection regression augmented with 1,000-fold cross-validation for internal validation was utilized. LASSO logistic regression is a logistic regression model that penalizes the absolute size of the coefficients of a regression model according to the value of λ. In the process of LASSO regression coefficients, some unimportant regression coefficients can be directly reduced to 0 to achieve the function of variable screening. In comparison to the ridge regression model, the penalty term in the LASSO regression is an absolute value, namely, L1 regular. The estimates of weaker factors shrink toward zero with larger penalties, then only the greatest predictors were left in the model. We select the most predictive covariates by the minimum value of λ. Subsequently, variables identified by LASSO regression analysis were used to construct the risk score with their coefficients: where n stands for the number of prognostic variables in the model; Value i is the original value of variable i ; and Coe i is the estimated coefficient of Value i in the LASSO logistic regression model. The probability of developing critical illness was calculated with the following formula: probability = exp We used receiver operating characteristic (ROC) curves to compare the sensitivity and specificity of scores generated with different machine learning techniques. The abscissa and ordinate coordinates of ROC curves are false-positive rate and true probability, respectively. The points of ROC curves reflect the susceptibility to the same signal stimulus. By comparing the falsepositive and true numbers, ROC curves show the performance of a classification model at all classification thresholds. The area under the receiver operating characteristics (AUROC), namely, the entire two-dimensional area underneath the entire ROC curve, was used as the precision measurement. AUROC shows how much the model is capable of distinguishing between classes. The larger the AUROC value, the better will be the model at predicting different classes. R-package "ROCR" was utilized for the calculation of the AUROC curve.
To explore temporal changes in laboratory findings during hospitalization, differences between critical illness groups during follow-up in laboratory findings were estimated from linear mixed models with R package "nlme." Details of samples used at each stage of statistical analysis were depicted in Figure 1. All statistical analysis was conducted with R software (version 3.6.2, R Foundation), and p-values were computed from two-tailed tests of statistical significance with a type I error rate of 5%.

External Model Validation
To validate the generalizability of the risk scores, we used an independent cohort from hospitals in Hunan province including 566 patients. We collected the same variables required for calculating the risk score from the validation cohort and cross-checked them. The 432 patients with at least 80% data completeness of the 28 variables used for model development were selected. The laboratory findings were imputed and the risk score was calculated as described for the development cohort. To assess the discriminative ability, the AUCs were evaluated.

Characteristics of the Cohorts
The development cohort with 1,363 patients, of which a total of 133 patients eventually developed critical illness (9.8%), from 4 hospitals in Hubei. The median follow-up time for patients was 14 days. The average (SD) age of patients in this cohort was 57.84 (16.29) years; 634 patients (46.52%) were men. The validation cohort included 566 patients with a mean (SD) age of 45.94 (15.33) years, 291 (51.41) were men. The median follow-up time for patients was 13 days. The critical illness eventually developed in 28 (4.24%) of these patients.

Prognostic Factors of Critical Illness
A total of 39 features were tested for associations with critical illness in the development cohort with univariate logistic regression analysis. The results of the 1,255 patients showed that 28 variables were prognosis factors for critical illness COVID-19 (p < 0.05, Table 1, Figure 2). The odds of critical illness were higher in patients older than 65 years. Laboratory results show that elevated CK-MB, neutrophils, PCT, α-HBDH, Ddimer, LDH, glucose, PT, APTT, RDW (SD and CV), fibrinogen, and AST were associated with a critical illness. Patients in the critical illness group showed lymphopenia and had a low rate of basophils, eosinophils, thrombopenia, red blood cell, hematocrit, hemoglobin concentration, and blood platelet count and represented decreased levels of K, Na, albumin, albumin to globulin ratio, and uric acid, compared with the non-critical illness group.

Longitudinal Observations of Laboratory Variables
To determine the major clinical features that appeared during COVID-19 disease progression, the dynamic changes in 28 clinical laboratory parameters were measured within 2 days after hospital admission and associated with critical illness, namely, hematological and biochemical parameters, were recorded from day 3 to day 25 after hospital admission. The temporal changes in laboratory findings during hospitalization were explored Frontiers in Public Health | www.frontiersin.org ( Figure 3). Baseline lymphocyte count was significantly lower in critical illness than in non-critical illness patients. Levels of CRP, D-dimer, LDH, and glucose were clearly elevated in the critical illness group compared with the non-critical illness group throughout the clinical course either in the developing dataset. Furthermore, we found that compared to that in the non-critical illness group, neutrophils, α-HBDH, and globulin were increased in the critical illness group, while eosinophils and albumin were decreased in the critical illness group.

Construction of the Risk Models and their Performances
A total of 28 variables determined at hospital admission and associated with a critical illness (Figure 2) were included in the model development. Prediction models were constructed using LASSO logistic regression, SVR, ANN, RT, and MARS, their performance was evaluated by the ROC analysis (Figure 4).
Although the predictive ability of ANN and SVR in the development cohort was better than other algorithms, the predictive ability using models of LASSO logistic regression and ANN outperformed the other algorithms in the validating dataset ( Figure 4D). The LASSO logistic regression model was selected by us for its high predictive power and interpretability.
In LASSO regression, after excluding irrelevant and redundant features (Figures 4A,B), 21 features remained for LASSO regression analysis, including age, whether take ARB drugs and blood test results, lymphocytes, neutrophils, blood platelet, thrombocytocrit, RDW (CV and SD), hematocrit, hemoglobin concentration, AST, CK-MB, albumin, LDH, glucose, K, Na, CRP, PCT, PT, APTT, fibrinogen, and uric acid. The risk score was constructed based on the coefficients from the LASSO logistic model ( Table 2) and then converted into a probability with formulas presented in the method and materials section. By internal 100 times bootstrap validation, the mean AUC based on data from the development cohort was 0.83 (95% CI, 0.78-0.86) ( Figure 4C). Variables utilized in the risk score for the validation cohort are shown in Table 3. The accuracy of the COVID risk score in the validation cohort was like that of the development cohort with an AUC in the validation cohort of 0.84 ( Figure 4D).

DISCUSSION
Early identification of patients with COVID-19 at risk of progression to critical illness disease will aid in better patient management and effective usage of healthcare resources. In this study, we unraveled that older age and higher levels of laboratory test indexes such as CRP, LDH, and glucose, and lower levels of laboratory findings such as lymphocytes and albumin on admission were associated with higher probabilities of critical illness COVID-19. In addition, a clinical risk score based on LASSO logistic regression was developed to predict the development of critical illness patients with COVID-19 with satisfactory accuracy according to AUC (0.83). Generally, the 21 variables required for estimating the probability of developing critical illness can be easily obtained from routine tests at hospital admission. The robustness and applicability of the risk score were confirmed in the independent validation dataset (AUC = 0.84). Univariate analyses revealed that factors, namely, age, neutrophils, D-dimer, LDH, CRP, glucose, APTT, fibrinogen, AST, and several other biochemical parameters were associated with a critical illness. In addition, the dynamic profile of the significant laboratory findings was tracked. Levels of LDH, Ddimer, glucose, CRP, α-HBDH, and globulin are higher in the critical illness group compared with the non-critical illness group. And neutrophil counts and albumin are lower in the critical illness group compared with the non-critical illness group. A prediction model for critical illness was developed with 21 predictors that were found to be independently correlated with critical illness via multivariate LASSO logistic regression analysis. Previous studies have found several of these variables to be prognosis factors for patients with COVID-19. It has been reported that elderly patients were more commonly critically ill with COVID-19 (3,26,27) and have a higher probability of a death outcome (28,29). Modelli and colleagues revealed that the 28-day fatality rate was associated with increasing age, hypertension, cardiovascular disease, and higher body mass index (17), in agreement with the previous work.
Lymphopenia, leukocytosis (with increased absolute neutrophil counts), eosinopenia, neutrophilia, increased CRP and PCT which reflects a persistent state of inflammation (30) may be related to cytokine storm and cellar immune deficiency induced by virus invasion (27,31). Zhou et al. found lower lymphocyte counts and higher LDH in patients who died from COVID-19 (1). Injured alveolar epithelial cells could lead to the infiltration of lymphocytes, resulting in persistent lymphopenia (32,33). Lymphopenia is a common characteristic in patients with COVID-19 and might play an important role in the disease process (34,35). Zhang et al. noted that 53% of patients admitted with COVID-19 had eosinopenia on the day of hospital admission (36). Calabrese et al. reported that lymphocyte and platelet counts were the most important features able to stratify patients into different clinical clusters (37). Ewan et al. demonstrated that risk stratification was improved by blood and physiological parameters (C-reactive protein, neutrophil/lymphocyte ratio, and neutrophil count) measured at hospital admission (20). Such findings were consistent with this work. A higher level of LDH was an indication of the activity and severity of idiopathic pulmonary fibrosis and is one of the most important prognostic biomarkers of lung injury (37). LDH was reported to be higher in severe and patients who received ICU treatment with COVID-19 than in mild and non-ICU patients (27,30,38,39), which is utilized as a valuable prognosis predictor (40,41). In addition, patients with elevated CK-MB levels on hospital admission were at significantly increased risk of critical illness. Li and colleagues found that cardiac injury (elevated LDH and CK-MB levels) were associated with severe disease or ICU admission and death in patients with COVID-19 (42). Increased PT and APTT, decreased blood platelet, thrombocytocrit, and fibrinogen which reflect the coagulation activation might be associated with the sustained inflammatory  response. Banoei et al. noted that prothrombin and lactate were the most differentiating biochemical markers in the mortality prediction model (18). Since hyperglycemia is harmful to the management of inflammation and viremia, the association between the level of glucose and critical illness in COVID-19 viral infections is not surprising. Based on big data analysis with a cohort with 7,337 COVID-19 cases, Zhu et al. revealed that diabetics with better-controlled blood glucose were associated with a decreased death risk than diabetics with poorly controlled blood glucose (43). Banoei and colleagues demonstrated that disease, coronary artery disease, dementia, age > 65, and altered mental status were the topmost differentiating mortality predictors (22).
Previous studies have identified that 15-53% of cases reported abnormal levels of AST during disease progression (44)(45)(46)(47). In a study conducted by Huang et al. (48), the elevation of AST was found in 8 (62%) of 13 patients in the ICU compared with 7 (25%) of 28 COVID-19 infected cases who did not need ICU care. Abnormal liver tests occur in most hospitalized patients with COVID-19 and may be associated with ICU admission, mechanical ventilation (48), and death (28,48). Liver damage (decreased albumin and increased globulin) in patients with COVID-19 infections might be associated with the direct effect of the viral infection of liver cells, drug hepatotoxicity, or immune-mediated inflammation (37), such as cytokine storm and pneumoniaassociated hypoxia.
Prediction models for the dynamic and control of COVID-19 infection found broad similarities with the features retained in our models, particularly regarding aging, hypertension, CRP, LDH, prothrombin, lactate, and neutrophil levels (14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24). The main advantage of the LASSO logistic regression is that the variable with a large parameter estimation is compressed to a smaller variable, while the variable with the smaller parameter estimate is compressed to 0. The parameter estimation of the LASSO analysis is continuous, which is suitable for model selection with high-dimensional data.
In the development dataset, we found that the discriminative abilities of SVR, ANN, RT, and MARS were outperforming that of LASSO logistic regression as evaluated by AUCs. However, in the independent validation dataset, the predictive ability of LASSO logistic regression was the best within all algorithms and was selected by us. The phenomenon that the model that incorporates the highest level of non-linearity displayed better in-sample prediction, but also yielded the worse out-ofsample performances may account for the over-fitting problem of the ANN, RT, MARS, and SVR algorithms (45). The linear Kernel function utilized in LASSO logistic regression performed badly in-sample but generated the best out-ofsample predictions.
There are inevitably limitations in our retrospective study. The primary one is incomplete laboratory findings in the electronic database and the lacking of CT images, which decreases the statistical power of the LASSO logistic regression model. Therefore, important information might be missed and further prospective studies are required. However, our model has a certain tolerance to missing data, as high performance as measured by AUC on the developing and external validation dataset for samples missing 20% of the predictors was achieved. Second, since the algorithms we tried are purely data-driven, the performances of these models may vary if developed with different datasets. We believe that more accurate models can be obtained with the increasing of available datasets. Third, the data for risk probability development and validation are from two provinces of China, which could potentially limit the generalizability of the risk model. Further studies on different populations all over the world with larger patient cohorts are needed to validate our findings.

CONCLUSION
In summary, this study identified 28 indicators (such as age, LDH, CRP, and lymphocytes) associated with critical illness of patients with COVID-19. The longitudinal laboratory variables were explored. A risk score to estimate the risk of developing critical illness among patients with COVID-19 was developed based on 21 variables independently associated with critical illness and commonly measured on hospital admission. The risk model is especially valuable for early detection and intervention of the incidence of critical illness COVID-19, thus making improvements to clinical strategies against COVID-19, optimizing the use of healthcare resources, and potentially reducing mortality in patients with COVID-19.

DATA AVAILABILITY STATEMENT
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

ACKNOWLEDGMENTS
We thank all study participants, their families, medical staff, and participating hospitals for their involvement and support in this study. We are grateful to the High Performance Computing Center of Central South University for assistance with the computations.