{'fraction_non_alphanumeric': 0.034536472473185274, 'fraction_numerical': 0.019760558992389515, 'mean_word_length': 5.076257436452137, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Application of interpretable machine learning for early prediction of prognosis in acute kidney injury

Graphical abstract


Background
Acute kidney injury (AKI), recognized as a serious disorder that acutely affects the kidney's filtration function, is very common in critically ill patients and has a high mortality rate [1][2][3]. A recent multinational cross-sectional study reported that AKI occurred in more than half of patients treated in the intensive care unit (ICU) [4]. Meanwhile, increasing AKI severity was strongly associated with increased mortality even after adjusting for possible confounders [4]. Despite the large number of new therapeutic strate-gies that have been conducted, no effective treatment has consistently demonstrated clinical benefits.
Early identification of AKI patients at high risk for clinical deterioration is of great importance and may help to deliver proper care and optimize the use of limited resources. Considering the potential benefits of electronic alerts in AKI, many researchers have developed a variety of machine learning-based models to predict mortality for AKI patients [5][6][7][8]. However, these previously established models were limited to clinical implementation due to the minimal interpretability and black box nature of the algorithms [9]. Opening the black box is crucial because it can allow clinicians to easily understand the internal logic of machine learning (ML) [10,11]. Recently, Explainable Artificial Intelligence (XAI) has been introduced to address the fundamental question about the rationale of the decision-making process in ML. The  (SHAP) [12] and Local Interpretable Model-Agnostic Explanations (LIME). These new interpretable methods have been successfully applied to explain the ML models related to mortality prediction in acute gastrointestinal bleeding [13] and sepsis [14], prediction of antimicrobial resistance [15] and the occurrence of AKI following cardiac surgery [16].
However, as far as we know, there has been relatively little analysis of the reliability and robustness of the explanation methods in outcome prediction among AKI. Therefore, in this study, we aimed to use a ML approach to predict in-hospital mortality in critically ill patients with AKI and utilize XAIs to increase the interpretability, fairness, and transparency of ML.

Ethics
The establishment of the Medical Information Mart for Intensive Care IV (MIMIC-IV version 1.0) database was approved by the Massachusetts Institute of Technology (Cambridge, MA) and Beth Israel Deaconess Medical Center (Boston, MA), and patients provided their consent to have their data captured in the database. Thus, the ethical approval statement was waived in this study, as the data in the MMIC-IV database were unidentifiable.

Data source
This study gathered clinical data from the MIMIC-IV database between 2008 and 2019. MIMIC-IV is a large, single-center, openaccess database comprising 76,540 ICU admissions [17]. We accessed the MIMIC-IV after completion of the Protecting Human Research Participants exam (Record ID: 47460147). This study was conducted in accordance with the principles of the Declaration of Helsinki in 2013, and all reporting followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Statement [18].

Study population
All adult (aged 18 years old and older) individuals diagnosed with AKI were enrolled in this study. AKI was defined and classified according to the Kidney Disease: Improving Global Outcomes (KDIGO) criteria during the first 24 h after ICU admission [19]. In brief, the KDIGO criteria included an increase in serum creatinine by 0.3 mg/dl within 48 h; an increase in serum creatinine to 1.5 times baseline, which is known or presumed to have occurred within the prior 7 days; or urine volume <0.5 mL/kg/hour for 6 h. Details on the definition and classification of AKI are provided in Supplementary Table S1. For patients with multiple ICU admissions during hospitalization, only the first admission was included for analysis. We excluded patients with an ICU length of stay of <3 h.

Data collection and preprocessing of data
We used structured query language (SQL) programming in Navicat Premium (version 15) to extract clinical data from the MIMIC-IV database. The information collected from the database followed the Deshmukh et al. [13] procedure. We collected information related to the patients' demographic characteristics, history of chronic diseases, vital signs, laboratory findings, medical treatments, severity scores of illness and outcomes.
Demographic variables collected for the study included age, sex, body weight and height. Medical conditions included hypertension, diabetes, congestive heart failure, cerebrovascular disease, chronic pulmonary disease, liver disease, tumor and acquired immune deficiency syndrome. We collected mean values in the first 24 h after ICU admission for the vital sign data, including heart rate, systolic blood pressure, diastolic blood pressure, mean blood pressure, respiratory rate, body temperature and SpO 2 . For laboratory findings, we collected the maximum value in the first 24 h after ICU admission including blood glucose, lactate, pH, pCO 2 , base excess, white blood cell count, anion gap, bicarbonate, blood urea nitrogen, serum calcium, serum chloride, serum creatinine, serum sodium, serum potassium, serum fibrinogen, international normalized ratio, prothrombin time, partial thromboplastin time, alanine aminotransferase, alkaline phosphatase, aspartate aminotransferase, total bilirubin, creative phosphokinase, creatine kinase MB and lactate dehydrogenase. The minimum values in the first 24 h after ICU admission were selected for the PaO 2 /FiO 2 ratio, hematocrit, hemoglobin, platelets, albumin and globulin. Medical treatments included the use of antibiotics, mechanical ventilation and vasopressors during the first 24 h after ICU admission. For the severity scores of illness, we calculated the maximum value for Sequential Organ Failure Assessment (SOFA), Oxford Acute Severity of Illness Score (OASIS), Simplified Acute Physiology Score II (SAPS-II), and the minimum value for Glasgow Coma Scale (GCS) during the first 24 h after ICU admission. We also collected the cumulative urine output during the first 24 h after ICU admission. A summary of each variable can be found in Supplementary Table S2. The code is available at https://github.com/MIT-LCP/mimic-iv.
Variables with more than 20% missing values were removed from further analysis. The multiple imputation method, recognized as a better approach to deal with missing observations in both outcome and independent variables, was used to handle missing data below 20% using the 'mice' package in R. To avoid overfitting, we used least absolute shrinkage and selection operator (LASSO) regression to identify potential variables associated with mortality.

Statistical analysis
The normal distribution of continuous variables was determined by the Kolmogorov-Smirnov test. The normally distributed variables were described as the means ± standard deviations (SD), while the skewed distributed variables were expressed as the median and interquartile range (IQR), and the categorical variables were expressed as number and percentages. Continuous variables between groups were compared by Student's t-test or the Mann-Whitney U test, as appropriate. Categorical variables between groups were compared by Pearson's chi-squared test or Fisher's exact test, as appropriate.
All statistical analyses were performed using Python (Version 3.6.6) and R software (Version 3.6.1, R Foundation for Statistical Computing). A P value (2-sided) below 0.05 was considered as statistically significant.

Machine learning model for feature mining and feature visualization
Stable and significant features were very important to predict the risks of in-hospital mortality, and the feature-related inhospital risks were studied with feature mining and feature visualization. In feature mining, all patients enrolled in the study were randomly split into a training set (75%) and a validation set (25%). We used the LASSO [20] algorithm to reduce the dimensionality of the features. The seven ML methods [Support Vector Machine (radial bias function) (SVM) [21], k-Nearest Neighbors (KNN) [22], eXtreme Gradient Boosting (XGBoost) [12], Decision Tree (DT) [23], Naive Bayes (NB) [24], Random Forest (RF) [25] and logistic regression (LR) [26] were used to develop and validate the models for assessing risks of in-hospital mortality. Predictive performance for the classifiers was evaluated using the accuracy, area under the receiving operating characteristic curve (AUC), sensitivity, and specificity measures. DeLong's test was performed to assess the differences in AUC. The calibration curve was used to compare the prediction probability of the models and the ground truth. Our final candidate model was selected based on the accuracy and the AUC. First, we used the SHAP values to visualize the significant features that influence the risks of mortality, to analyze the importance of individual features affecting the output of the model and to visualize the impact of key features on the final model in individuals. Next, we conducted the LIME [27] algorithm to fit the predictive behavior of the model. The details of the pro-cedure in this study are shown in Fig. 1. Subgroup analyses based on different KDIGO stages were performed. Sensitivity analyses were conducted for the outcome by removing the most crucial factor from our final candidate model.

Participants
A total of 33,020 participants diagnosed with AKI were screened for eligibility; of these 33,020 patients, 10,657 were excluded due to multiple ICU admission (only the first admission was included for analysis), and 3 were excluded due to an ICU length of stay of <3 h. Finally, 22,360 patients were eligible for participation (Fig. 1). The prevalence of in-hospital mortality was 15.6% (3484/22360). These patients had a median age of 69.5 (IQR, 58.3-79.9) years and 42.8% (9574/22360) were female. Hypertension (9245/22360, 41.3%), diabetes (7268/22360, 32.5%) and congestive heart failure (6819/22360, 30.5%) ranked as the top 3 comorbidities. The baseline characteristics of the dataset are summarized in Table 1.

Predictor selection
Nineteen variables with more than 20% missing values were removed ( Supplementary Fig. S1). Multiple imputation by chained equation was used to impute missing values for each variable below 20%. Forty variables measured on the first day after ICU admission were included in the LASSO regression. After LASSO regression selection with 5-fold cross-validation via minimum criteria, 29 variables remained as significant predictors for mortality ( Supplementary Fig. S2), and the details of the variables are shown in Supplementary Table S3.

Model development and validation
A total of 22,360 patients were selected and randomly split into a training cohort (16770, 75%) and a validation cohort (5590, 25%). We used 29 variables selected by LASSO regression as input factors (Supplementary Table S3), and seven ML methods, including SVM, KNN, XGBoost, DT, NB RF and LR, were established to predict outcome. In the validation cohort, the XGBoost model achieved the  Fig. 2A).
To further evaluate the performance of the seven models, precision, sensitivity and specificity were also calculated and the results are shown in  (Fig. 2B). Thus, the XGBoost model was selected for further prediction.

Model explainability
We tried to open the 'black box' in the XGBoost model by SHAP values and explained how the model works in predicting mortality. The feature importance ranking with the SHAP summary plot for the XGBoost model is presented in Fig. 3A, and the top 4 most important variables contributing to the model were GCS, blood urea nitrogen, cumulative urine output on Day 1, and age. Additionally, we depicted how a single variable affected the output of the XGBoost prediction model using SHAP dependence analysis (Fig. 3B). More detailed results of the top 4 most important clinical features affecting the output of the XGBoost prediction model are shown in Fig. 4.
Then, we used SHAP force analysis and the LIME algorithm to explain the individualized prediction of death by drawing two samples from the validation set. Fig. 5A and Fig. 5B present a deceased case using SHAP force analysis and the LIME algorithm, respectively. This case was an 81-year-old woman with a history of congestive heart failure and chronic pulmonary disease admitted to the ICU for AKI. The predicted probability for death by the XGBoost model was 93%. Factors detected by the XGBoost model that increased the risk of death were GCS score of 3, urine output on Day 1 of 162 mL, blood urea nitrogen of 45 mg/dL and serum sodium of 154 mmol/L. Factors that decreased the risk of death included the lack of a history of cerebrovascular disease or tumor. The outcome predicted by the XGBoost model was death for this patient and the actual outcome was also death. Similarly, Fig. 5C and Fig. 5D present a survival case using SHAP force analysis and the LIME algorithm, respectively. This case was a 79-year-old woman admitted to the ICU for AKI. The predicted probability for death by the XGBoost model was 17%. The patient's elevated serum sodium of 150 mEq/L, a need for mechanical ventilation on Day 1 and systolic blood pressure of 102 mmHg contribute to increasing the death risk, while a GCS score of 14, the cumulative urine output of 1790 mL on Day 1 and no history of tumor or cerebrovascular disease contributed to decreasing the patient's death risk. The outcome predicted by the XGBoost model was survival for this patient, and the actual outcome was also survival.

Subgroup analyses
Prespecified subgroup analyses conducted for the different KDIGO stages showed that the XGBoost model remained robust in predicting mortality among patients with KDIGO stage 1   Supplementary Fig. S4.

Sensitivity analyses
When excluding the GCS score in the XGBoost model, the AUC for mortality prediction in AKI patients was 0.830 ( Supplementary  Fig. S5). In addition, the prediction results of the XGBoost model on samples were counted in the validation set, and the distribution of the two cases (one survival and one deceased) of the aggregated prediction results was also visualized using the LIME algorithm ( Supplementary Fig. S6).

Discussion
In this study, we developed and validated seven ML methods using twenty-nine clinical variables to assess the risks of inhospital mortality in critically ill patients with AKI. The XGBoost model exhibited the best performance in terms of discrimination and accuracy. Subgroup analyses also revealed that the XGBoost model achieved robust performance for the prediction of death in different KDIGO stages. Additionally, SHAP values were used to reveal the feature importance and how particular compound substructures influence the XGBoost prediction. GCS, blood urea nitrogen, cumulative urine output on Day 1 and age were the top 4 most important variables contributing to the XGBoost model. Finally, the LIME algorithm was used to facilitate the individualized predictions.
Accurate and timely prediction of mortality for AKI is required to identify patients at high risk of clinical deterioration so that preventive measures can be taken in a timely manner, which may reduce mortality. Several studies have attempted to establish prognostic models among AKI patients with ML methods and showed a modest prognostic yield [5,7,8,[28][29][30][31][32][33][34]. For example, a study from the US constructed a prognostic model for predicting 60-day mortality in critically ill patients with AKI. They found that the predictive model with logistic regression yielded an AUC of 0.85 (95% CI: 0.83-0.88), surpassing those of APACHE II and SOFA [5]. However, this study did not perform internal or external validation. In 2019, Lin and colleagues constructed prediction models of mortality risk based on the RF, ANN (artificial neural network) and SVM for AKI patients. In the testing set, they found that the RF model had the largest AUC (0.866, 95% CI: 0.862-0.870) [7]. A recent modeling study for AKI also found that the XGBoost model achieved the best performance with an AUC of 0.796, compared to the LR, SVM and RF models (AUC: 0.662, 0.667, and 0.692, respectively) [28]. However, all the above ML models were established based on limited algorithm tools and were at a loss to adequately explain how they work. In the current study, we developed a variety of ML approaches containing the linear model (e.g., LR), kernel-based method (e.g., SVM), gradient boosting classifier (e.g., XGBoost) and other ML models (e.g., KNN, NB, DT and RF) and selected the optimal one with the best performance in discrimination and accuracy. Furthermore, we summarized the previous clinical prediction model related to mortality risk prediction in AKI patients and found that the XGBoost model in the current study performed best with an AUC of 0.890 in the validation cohort (Supplementary  Table S5). Moreover, given the opaque black box nature of ML, we utilized the SHAP values and LIME algorithm to interpret and reveal the most important factors contributing to the prediction results, which greatly improved the model's interpretability.
In the current study, a summary of feature importance in the XGBoost model showed that the GCS score was the most crucial factor in the development of death in patients with AKI. The GCS score, as a common indicator of consciousness ranging from 3 to 15, can obviously show the neurological changes among critically ill patients [35]. Concurrently, there was a strong correlation between the GCS score and clinical outcome [36]. Nevertheless, none of the previous models included this critical feature in predicting mortality risk for AKI patients [5,7,8,28,29]. Our study found that blood urea nitrogen (ranked 2nd) and the cumulative urine output on Day 1 (ranked 3rd) were closely associated with mortality in AKI. Blood urea is excreted mainly by the kidney. The elevated blood urea nitrogen level indicated renal lesions, which also increased the risk of adverse outcomes [4]. Additionally, urine output has been found to be a marker for AKI and to be associated with adverse outcomes in critically ill patients [37]. Age was also another key factor for predicting mortality, which had been studied in previous reports as an independent predictor for hospital mortality among AKI patients [8].
The present study developed and validated a good performance model for mortality prediction in critically ill patients admitted for AKI, with a large sample size from the MIMIC-IV database and a precise definition of AKI. The model was built based on 29 candidate variables related to patients' demographic characteristics, medical history, vital signs, laboratory findings, medical treatments and GCS score, which was more systematic and robust than previously reported models that included only parts of investigative modalities. Overfitting is one of the critical problems in developing models by ML, which may lead to inadequate conclusions. In this study, we performed LASSO regression to reduce overfitting in the process of feature selection. This method surpasses the method of selecting predictors according to the strength of their univari-able association with outcome. In addition, no other severity score of illness except GCS was enrolled as input variables for model construction due to the possible reuse of variables (such as SOFA score contains total bilirubin, vasopressor use, GCS, serum creatinine and urine output). Additionally, we attempted to use the latest methods (SHAP value and LIME algorithm) to interpret and analyze why and how the XGBoost model worked during execution. Although LIME and SHAP values showed a similar trend overall for model interpretation, they were also presented with distinguishing specific regions of slight mismatch. These differences indicated where features go from favoring survived to favoring deceased. Moreover, this early warning system is being translated for clinical implementation in cooperation with technology companies, and our team will look forward to its clinical use in the future.
However, there were also several limitations in this study. First, many laboratory parameters were removed before model construction due to missing data of over 20% (e.g., D-dimer, globulin, and albumin). Second, as a critical parameter to understand the pathological mechanisms of disease, the etiology of AKI was not recorded in the MIMIC-IV database, and we failed to add this factor to the model construction. These may have inevitably caused selection bias. Third, the clinical indicators were gathered using a single-center database due to the nature of MIMIC-IV. Fourth, although XGBoost does take into account interactions between fea- tures, these potentially relevant interactions will not be shown by LIME as it is a linear model. In addition, although LIME provides an explanation, this explanation depends on the parameters used to develop the local model, and by changing the parameters, other (potentially very different) explanations can arise. Fifth, the models were validated internally only in this study and further multicenter external validation is needed to verify the model's discriminating ability and generalizability.

Conclusions
The ML models based on clinical features were developed and validated with great performance in the early prediction of a high risk of death in AKI. Application of the SHAP values and LIME algorithm in ML may help physicians in clinical decision-making.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Funding
This work was supported by the National Natural Science Foundation of China (grants 81772046 and 81971816 to Dr. Peng and 11831015 to Dr. Zou) and the Special Project for Significant New Drug Research and Development in the Major National Science and Technology Projects of China (2020ZX09201007 to Dr. Peng). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.