{'fraction_non_alphanumeric': 0.04248704663212435, 'fraction_numerical': 0.023794340374651255, 'mean_word_length': 5.172447724477244, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Machine learning-based prognostic modeling using clinical data and quantitative radiomic features from chest CT images in COVID-19 patients

Objective To develop prognostic models for survival (alive or deceased status) prediction of COVID-19 patients using clinical data (demographics and history, laboratory tests, visual scoring by radiologists) and lung/lesion radiomic features extracted from chest CT images. Methods Overall, 152 patients were enrolled in this study protocol. These were divided into 106 training/validation and 46 test datasets (untouched during training), respectively. Radiomic features were extracted from the segmented lungs and infectious lesions separately from chest CT images. Clinical data, including patients’ history and demographics, laboratory tests and radiological scores were also collected. Univariate analysis was first performed (q-value reported after false discovery rate (FDR) correction) to determine the most predictive features among all imaging and clinical data. Prognostic modeling of survival was performed using radiomic features and clinical data, separately or in combination. Maximum relevance minimum redundancy (MRMR) and XGBoost were used for feature selection and classification. The receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), sensitivity, specificity, and accuracy were used to assess the prognostic performance of the models on the test datasets. Results For clinical data, cancer comorbidity (q-value < 0.01), consciousness level (q-value < 0.05) and radiological score involved zone (q-value < 0.02) were found to have high correlated features with outcome. Oxygen saturation (AUC = 0.73, q-value < 0.01) and Blood Urea Nitrogen (AUC = 0.72, q-value = 0.72) were identified as high clinical features. For lung radiomic features, SAHGLE (AUC = 0.70) and HGLZE (AUC = 0.67) from GLSZM were identified as most prognostic features. Amongst lesion radiomic features, RLNU from GLRLM (AUC = 0.73), HGLZE from GLSZM (AUC = 0.73) had the highest performance. In multivariate analysis, combining lung, lesion and clinical features was determined to provide the most accurate prognostic model (AUC = 0.95 ± 0.029 (95%CI: 0.95–0.96), accuracy = 0.88 ± 0.046 (95% CI: 0.88–0.89), sensitivity = 0.88 ± 0.066 (95% CI = 0.87–0.9) and specificity = 0.89 ± 0.07 (95% CI = 0.87–0.9)). Conclusion Combination of radiomic features and clinical data can effectively predict outcome in COVID-19 patients. The developed model has significant potential for improved management of COVID-19 patients.


Introduction
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) disease (COVID-19) has significantly impacted global health and continues to be a major global concern as the number of infected patients and mortality are still rapidly growing [1,2]. The first line approach to diagnose COVID-19 involves the usage of a molecular diagnostic method, referred to as real-time quantitative reverse transcription-polymerase chain reaction (qPCR) assay [3]. In addition, X-ray computed tomography (CT) has garnered much clinical and research interest for the management of COVID-19 patients [4]. Several studies have compared the two diagnostic methods and documented their benefits and limitations. For instance, some of these studies reported that qPCR has variable sensitivity for different biological samples, while CT was unable to detect small infected lung regions [3][4][5]. Furthermore, the importance of predicting patients' prognosis based on early findings in the course of the disease has been an area of active research [6,7].
The emerging field of radiomics provides a reliable, non-invasive and cost-effective approach to improve diagnosis, prognosis and therapy response prediction in a number of diseases [8][9][10][11][12][13][14][15]. Radiomics is an image data mining framework enabling to extract extensive information from medical images using a wide range of features, based on which a correlation is established with clinical and biological findings [8,9,[16][17][18][19][20][21][22]. Furthermore, radiomic studies can be used to provide differential diagnosis [23]. CT image radiomics are increasingly utilized for this purpose. Yanling et al. [24] developed a radiomics nomogram incorporating CT radiomic signatures and laboratory data for differentiating bacterial pneumonia from acute paraquat lung injury. In another study, Wang et al. [25] applied CT radiomics for differential diagnosis of progressive pulmonary tuberculosis from community-acquired pneumonia. Their radiomics model outperformed senior radiologists' clinical judgment [25].
A number of studies applied deep or machine learning algorithms for COVID-19 outbreak prediction, detection/segmentation of infected pneumonia regions from radiologic images, as well as new drug development and disease screening [26][27][28][29][30][31][32][33][34][35]. In diagnostic studies, artificial intelligence approaches have been applied to various medical imaging modalities, including radiography, ultrasound, and CT to build more accurate detection/diagnostic models [36,37]. For the specific case of CT, a number of radiomic studies have been conducted for detection, including screening patients from other lung infections, and prediction of hospital stay. In these studies, CT radiomic features and machine learning algorithms were used to develop and implement such models. Qi et al. [4] studied 52 COVID-19 patients for predicting hospital stay. CT radiomic features and machine learning algorithms, including logistic regression and Random Forest were employed, wherein the model exhibited area under the receiver operating characteristic (ROC) curve (AUC) values of 0.97 and 0.92 for logistic regression and Random Forest algorithms, respectively. The detection radiomic models developed by Guiot et al. [38] depicted a sensitivity and specificity of 78.9% and 91.1%, respectively, whereas the radiomics signature to detect COVID-19 from CT images developed by Fang et al. [39] achieved AUC of 0.82 for the test sets.
When utilizing radiomics and machine learning or deep learning approaches, studies have indicated that these approaches alone or in combination with clinical information have the potential to serve as substitutes for diagnosis, prognosis and therapy response performance evaluation [9,11]. In the present study, we aimed to develop prognostic models to predict survival (alive or deceased status) in COVID-19 patients. Specifically, we aimed to develop various prognostic models using CT radiomic features, clinical data (demographics and history, and laboratory tests) and radiological scores obtained from radiologist's reports. Fig. 1. Flowchart of the adopted study protocol.  Table 1.

Patient population
This retrospective study was conducted with institutional review board approval. Formal written consent was waived owing to the nature of the study. All patients admitted to our tertiary center between February 11, 2020 and Jun 20, 2020 were enrolled in our study protocol. First, we collected a COVID-19 dataset by applying a set of inclusion and exclusion criteria. Our inclusion criteria were as follows: 1) patients undergoing high quality CT scans, 2) confirmation of COVID-19 by qPCR, 3) visible infected regions in the lungs, and 4) availability of clinical and radiological data and reports. The inclusion and exclusion criteria of patients are presented in Fig. 2. All patients had a median of 6 days interval between symptoms start date and admission to the hospital. All patients received one of the two standard treatment regimens administered in the hospital: Hydroxychloroquine, Lopinavir/Ritonavir.  All clinical, laboratory and imaging features were extracted at the first day of admission.

Laboratory data
Upon admission, the results of laboratory tests were extracted from patient medical records. These included aspartate aminotransferase in U/L (AST), alanine aminotransferase (ALT) in U/L, alkaline phosphatase (ALP) in U/L, total and direct bilirubin (T.Bill and D.Bill, respectively) in mg/dL, hemoglobin (HB) in g/dL, white blood cells (WBC) in/mm 3 , venous blood gas analysis of acidity (PH), carbon dioxide concentration (PCO2), and bicarbonate concentration (HCO3), C-reactive protein in mg/L (CRP), platelet count in/mm 3 (Plt), blood creatinine level in mg/ dL (Cr), blood urea nitrogen (BUN) in mg/dL, prothrombin time (PT) in seconds, partial thromboplastin time (PTT) in seconds, prothrombin time normalized with the international normalized ratio (INR), procalcitonin levels (PCT) in ng/dL, and sodium and potassium (Na and K, respectively) in mEq/L. We also used the differential counts of neutrophils, lymphocytes, monocytes, and eosinophils in percentages (Neutr. Diff, Lymph.Diff, Mono.Diff, and Eosin.Diff, respectively) [6,[41][42][43].

CT imaging
CT scanning with breath holding was performed on a 16 detectorrow Brilliance 16CT scanner (Philips Medical Systems, Best, the Netherlands) using the following scanning parameters: A tube voltage of 100 KVp was used for patients with BMI ≤ 30 (111 patients) whereas 120 KVp was used for patients with BMI > 30 (41 patients); 45 mA tube current; 16 × 1.5 mm collimation; 0.5 s rotation time; pitch of 1.0; and 35 cm field-of-view [6,[41][42][43]. Low quality CT images owing to patient bulk motion, severe respiratory motion, or when the axial coverage was less than the total lungs were excluded.

Image preprocessing and feature extraction
All CT images were interpolated to isotropic voxel and re-sampled to 1 × 1 × 1 mm 3 [48]. Subsequently, bin discretization to 64gray levels were performed for radiomics analysis [48]. For feature extraction, first-order statistics (19 FOS features), shape-based (16 shape features), gray-level co-occurrence matrix (GLCM 23 features), gray-level run length matrix (16 GLRLM features), gray-level size zone matrix (16 GLSZM features), neighboring gray tone difference matrix (5 NGTDM features), and gray level dependence matrix (14 GLDM features) features were extracted [48,49]. Feature extraction was performed using PyRadiomics [49] v2.1.2 python-based software package, which was standardized through the Image Biomarker Standardization Initiative (IBSI) [48]. Full details about the feature categories are provided in Table 1. We also constructed 15 new shape features through division of lesion shape features by whole lung shape features to extract relative shape features.

Univariate analysis
We performed univariate analysis after normalization of each feature to Z-scores to determine the prognostic importance of each feature (clinical, radiological and radiomics). For continuous features, we performed Student's t-test and area under the ROC curve (AUC) analysis. The performance of categorical features was evaluated using the Chisquare test. We also applied false discovery rate (FDR) correction to the q-value (FDR adjusted p-value) to assess the significance of the features.

Multivariate machine learning analysis
The maximum relevance minimum redundancy (MRMR) algorithm [50] was used for feature selection. To this end, the maximum-relevance selection approach was employed to select features with maximal correlation to patients' outcome (alive or deceased status) and minimum-redundancy selection, thus ensuring minimal redundancy  among features [50]. The eXtreme Gradient Boosting (XGBoost) machine learning algorithm [51] which is an ensemble learning algorithms based on different decision trees was adopted for classification. Feature selection and classification were performed using praznik and caret R packages, 2 respectively.

Data modeling and univariate analysis
We developed various prognostic models using the collected data and the adopted classification method. Our models were: 1) Clinical (pre-clinical, lab and radiological data), 2) Lung radiomics (radiomic features extracted from the whole lung), 3) Lesion radiomics (radiomic features extracted from lesions), 4) Lung + Lesion (combined radiomic features extracted from whole lung and lesions), 5) Lung + Clinical (combined radiomic features extracted from whole lung, clinical and radiological data), 6) Lesion + Clinical (combined radiomic features extracted from lesions, clinical and radiological data) and 7) Lung + Lesion + Clinical (combined radiomic features extracted from whole lung, lesions, clinical and radiological data). For model validation, 106 patients were used as the training/validation dataset whereas the remaining 46 patients were used as the test (unseen and untouched during training) dataset. The ROC, AUC, accuracy (ACC), sensitivity (SEN) and specificity (SPE) were used to assess the prognostic performance of the models. The different steps followed are summarized in Fig. 3: 1. The model was trained using 106 patients' data. Yet, to find the optimal hyperparameters of models, we used bootstrap resampling with 1000 repetitions. Bootstrap techniques were used for XGBoost hyperparameters tunning (using the random search method) implemented with 1000 repetitions. 2. After tuning and optimizing the models based on AUC, we selected the optimal model. 3. The optimal model was tested on the test set (test datasets untouched/unseen during bootstrapping). We calculated Accuracy, AUC, Sensitivity, and Specificity for the optimal model on the test datasets. 4. Steps 1-3 were repeated 100 times to ensure the repeatability of the results. 5. The Mean, SD and CI95% were calculated from step 4 for Accuracy, AUC, Sensitivity, Specificity metrics. 6. Clinical, radiomics and combined models were statistically evaluated using the results of step 5.
We repeatedly trained a bootstrapped model with 1000 repetition (on 106 patients dataset) and tested on an independent dataset for 100 times to make sure that the results are repeatable for different models. All results were reported on 46 test sets (unseen during model training by bootstrap) by 100 times repetitions. The mean, standard deviation and 95% confidence interval (CI) were reported for each model for 100 times repetitions of the whole process. After data normality test using Kolmogorov-Smirnov normality test, we used Wilcoxon signed-rank test to determine significant differences between the models. A p-value < 0.05 was used as a criterion for statistically significant differences. All statistical analysis was performed using R 3.6.3 software.

Patient population
Following application of inclusion and exclusion criteria, 152 patients including 87 males and 65 females were retained from an initial triage of 545 patients. Forty patients with a mean age of 65.7 years had critical conditions and eventually died, whereas 112 cases with a mean age of 59.5 years fully recovered from COVID-19. The flowchart in Fig. 2 shows the number of excluded, included, recovered and deceased patients. The details of the descriptive statistics of continuous and categorical features of patients are presented in Tables 2 and 3 for the training/validation and test sets (unseen and untouched during training).

Univariate analysis
Our univariate analysis of clinical features is shown in Supplemental  Figures 1 and 2 as categorical and continuous features in terms of AUC,   Table 5 Confidence interval (CI) of area under the curve (AUC), accuracy (ACC), sensitivity (SNE) and specificity (SPE) in the test set for the different models. Amongst continuous clinical features, systolic blood pressure, diastolic blood pressure, hemoglobin, platelet, lymphocyte, neutrophil, monocyte, had a q-value < 0.05. Amongst discrete (categorical) clinical features, smoking, cancerous, consciousness and total involvement score proved to be statistically significant features (p-value < 0.05) between two alive and deceased group. For univariate radiomics analysis, the results are displayed in Supplemental Figures 3 and 4. Amongst lung radiomic features, SAHGLE (AUC = 0.70) and HGLZE (AUC = 0.67) from GLSZM, JA and SA from GLCM, Busyness from NGTDM and Median from FO (AUC = 0.67) were found as most prognostic features. Amongst lesion radiomic features, RLNU from GLRLM, HGLZE from GLSZM, DNU from GLDM, Range from FO and Volume from Shape (AUC = 0.73) had the highest performance with significant q-value after FDR correction.

Models
Importance features selected by the MRMR algorithm were reported for each model in Supplemental Table 2. Fig. 4 depicts the heat map of AUC, ACC, SEN and SPE for different combinations of models. The mean (STD) and confidence interval (CI) for AUC, ACC, SEN and SPE for the developed models (test set) are summarized in Tables 4 and 5, respectively. Our results indicated that the combined model (Lung + Lesion + Clinical) had the highest prognostic capability with AUC = 0.95 ± 0.02, ACC = 0.88 ± 0.04, SEN = 0.88 ± 0.06 and SPE = 0.89 ± 0.07. The 95% CI for these parameters were 0.95-0.96, 0.88-0.89, 0.87-0.90 and 0.87-0.90, respectively.
The ROC curve and boxplot of these models for the test set are shown in Figs. 5 and 6, respectively. In the boxplots, significant differences among the models can be observed. To compare the models in terms of significant changes in AUC, ACC, SEN and SPE, p-value plots are shown in Fig. 7. It can be seen that the combined Lung + Lesion + Clinical model has significant AUC differences (p < 0.05) relative to other models. The model was also significantly different in terms of ACC with respect to all models. With respect to SPE, all models had significant differences, except Lung + Clinical model, whereas for SEN, all models were significantly different, except the Lesion + Clinical model. In terms of AUC, except Lung and Lesion, Lung + Clinical and Lung, Lung + Lesion and Lung models as well as Lung + Clinical and Lesion models were significantly different (p < 0.05).

Discussion
A novel approach for prognostication of COVID-19 patients using different image-derived features, including semantic, radiomics and clinical data (demographics and history, laboratory tests and visual scoring of CT by radiologists) was presented in this work. We demonstrated that clinical and quantitative radiomic features, alone or in combination, can be used as potential biomarkers for the prediction of survival in COVID-19 patients. Although some radiomic studies have been conducted in the framework of COVID-19, this is to the best of our knowledge, the first study reporting on the use of advanced combined models for prognosis survival analysis.
We extracted two categories of radiomic features from whole lung and lung lesions. The aim was to assess how the extracted radiomic features might be used as different prognostic parameters. Decoding heterogeneity is among the aims of radiomics analysis. Hence, since the delineated lesions and whole lungs have different characteristics, they could serve as different markers. Conversely, combining these features along with other clinical parameters provides more variables for developing more predictive models. In previous lung radiomic studies, adding clinical data to radiomic signatures improved model performance. A study by Chen et al. [52] demonstrated that adding clinical data, such as smoking history, enabled to slightly improve performance for differentiating peripherally-located small cell lung cancer from non-small cell lung cancer using CT radiomic features.
Univariate analysis showed that some clinical parameters might be predictive. Although for single clinical parameters, the highest AUCs were achieved for BUN, oxygen saturation, monocyte count and a number of involved zones. These findings were congruent with previous studies. Several studies have suggested that increased BUN can be attributed to acute kidney injury during the course of the disease, which can be a cause of adverse effects [53][54][55]. Colombi et al. [56] reported lung involvement area and hypoxia as reliable predictors of ICU admission and mortality among COVID-19 patients. Conversely, we believe that the fact that the results on monocyte count can be an outcome predictor is the consequence of our sample size as Zeng et al. [57] did not report significant differences between severe and non-severe patients in a population of 3090 patients from 15 different studies. With respect to single imaging measures, we observed that several radiomic features were predictive in both lesion and whole lung delineations. When comparing these two feature categories, it appeared that whole lung features have higher AUCs. A recent study by Tan et al. [58] demonstrated the predictive value of non-focus area of CT images to distinguish different clinical types of COVID-19 pneumonia. In our study, whole lung features provided more relevant characteristics of the disease in COVID-19 patients. Extracting radiomic features from infected and non-infected regions provided a more accurate prognostic model.
On the modeling of prognosis prediction, we observed that the combined model including all measures had the highest performance and had significant differences with other models. In addition, we showed that other models have similar behavior, although there were significant differences among them. For both radiomic models, the AUC had a range varying from 0.91 to 0.93. However, when they were combined with clinical features, the AUC improved from 0.91 to 0.93 to 0.95-0.96. These results indicated that the combined models provided more accurate information. In addition, compared to univariate radiomics analysis, multivariate modeling results in more reliable results. It should be emphasized that in predictive modeling, the variables are the heart of the model, although the classifiers have a critical role [59,60]. In this work, we used a classifier with a wide range of features.
A wide range of image analysis algorithms combined with machine learning techniques were recently designed for COVID-19 detection, diagnosis and prognosis. Li et al. [61] applied artificial intelligence algorithms to distinguish COVID-19 from community acquired pneumonia on chest CT. They developed a deep learning model, COVID-19 detection neural network (COVNet), to extract visual features from volumetric chest CT examinations for the detection of COVID-19 and compared them to community acquired pneumonia and other non-pneumonia images. Their model achieved a sensitivity and specificity of 90% and 96%, respectively, with an AUC of 0.96. Although deep learning models have achieved a high predictive/prognostic performance, their mechanisms of action are not fully understood [62]. Yet, radiomic features may provide more reliable results because they capture tissue characteristics. Hence, studies have indicated that these markers could decode the biological properties of the tissues [63,64]. In this regard, we believe that our results could be exploited reliably in clinical practice.
Although the presented results are important, this study inherently bears a number of limitations. First, the sample size is low and there is a lack of external validation set from different centers. Further clinical studies are needed to verify our results with larger clinical databases. Second, therapeutic strategies for patients were not considered in this study. Including treatment parameters in the models will provide more reliable results. Third, although our imaging and radiomics settings were similar for all patients, we suggest to assess radiomic features reproducibility before clinical adoption. Image segmentation was performed by an experienced radiologist only once, and as such, it was not possible to estimate the intra-observer variability and repeatability of the segmentations. Forth, we only tested one feature selection and classifier algorithms. As there is no one-fits-all machine learning algorithm, different combinations of feature selectors and classifiers result in different performance. Future studies should focus on evaluating different algorithms and comparing their performance [65,66]. Various inclusion and exclusion criteria were applied retrospectively to the datasets, which decreased the number of cases. Patients with severe motion artifacts, which is inevitable in some non-cooperative or unstable patients, were excluded. This might impact the generalizability of the obtained results. Future studies should use a more heterogeneous dataset in terms of CT image quality to ensure the generalizability of the model.

Conclusion
We demonstrated that the combination of radiomic features, clinical and radiological data could be used to effectively predict survival in COVID-19 patients. To the best of our knowledge, this is the first study applying such methodology for COVID-19 prognosis survival modeling. We also demonstrated that there are a number of individual predictive clinical or imaging features having the potential to be used in routine clinical practice for more accurate management of COVID-19 patients.

Declaration of competing interest
The authors declare that they have no conflict of interest.