{'fraction_non_alphanumeric': 0.02747557592570257, 'fraction_numerical': 0.015366059582680015, 'mean_word_length': 5.226494442775608, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Survival prediction in intensive-care units based on aggregation of long-term disease history and acute physiology: a retrospective study of the Danish National Patient Registry and electronic patient records

Summary Background Intensive-care units (ICUs) treat the most critically ill patients, which is complicated by the heterogeneity of the diseases that they encounter. Severity scores based mainly on acute physiology measures collected at ICU admission are used to predict mortality, but are non-specific, and predictions for individual patients can be inaccurate. We investigated whether inclusion of long-term disease history before ICU admission improves mortality predictions. Methods Registry data for long-term disease histories for more than 230 000 Danish ICU patients were used in a neural network to develop an ICU mortality prediction model. Long-term disease histories and acute physiology measures were aggregated to predict mortality risk for patients for whom both registry and ICU electronic patient record data were available. We compared mortality predictions with admission scores on the Simplified Acute Physiology Score (SAPS) II, the Acute Physiologic Assessment and Chronic Health Evaluation (APACHE) II, and the best available multimorbidity score, the Multimorbidity Index. An external validation set from an additional hospital was acquired after model construction to confirm the validity of our model. During initial model development data were split into a training set (85%) and an independent test set (15%), and a five-fold cross-validation was done during training to avoid overfitting. Neural networks were trained for datasets with disease history of 1 month, 3 months, 6 months, 1 year, 2⋅5 years, 5 years, 7⋅5 years, 10 years, and 23 years before ICU admission. Findings Mortality predictions with a model based solely on disease history outperformed the Multimorbidity Index (Matthews correlation coefficient 0⋅265 vs 0⋅065), and performed similarly to SAPS II and APACHE II (Matthews correlation coefficient with disease history, age, and sex 0·326 vs 0·347 and 0·300 for SAPS II and APACHE II, respectively). Diagnoses up to 10 years before ICU admission affected current mortality prediction. Aggregation of previous disease history and acute physiology measures in a neural network yielded the most precise predictions of in-hospital mortality (Matthews correlation coefficient 0⋅391 for in-hospital mortality compared with 0⋅347 with SAPS II and 0⋅300 with APACHE II). These results for the aggregated model were validated in an external independent dataset of 1528 patients (Matthews correlation coefficient for prediction of in-hospital mortality 0⋅341). Interpretation Longitudinal disease-spectrum-wide data available before ICU admission are useful for mortality prediction. Disease history can be used to differentiate mortality risk between patients with similar vital signs with more precision than SAPS II and APACHE II scores. Machine learning models can be deconvoluted to generate novel understandings of how ICU patient features from long-term and short-term events interact with each other. Explainable machine learning models are key in clinical settings, and our results emphasise how to progress towards the transformation of advanced models into actionable, transparent, and trustworthy clinical tools.


Introduction
Intensive-care units (ICUs) handle patients from all medical and surgical specialties. Therefore, their populations are highly heterogeneous, and consist of mainly elderly patients who often have a long history of disease. Prediction of prognosis to inform decision making in the ICU is difficult because of the severity of patients' current illness and their disease history. 1 Mortality risk estimates based on acute physiology scores-such as the Simplified Acute Physiology Score (SAPS) and the Acute Physiologic Assessment and Chronic Health Evaluation (APACHE)-are sometimes used in clinical practice to assess disease severity. 2,3 They are based on logistic regression of specific markers of patient physiology that are recorded during the first hours after ICU admission. In the past 10 years, advanced modelling and machine learning techniques have shown promising results in improving prediction of ICU prognosis based on acute physiology. 4 The extent to which disease events before ICU admission affect prognosis has been debated. 1,5 An increasing number of chronic comorbidity categories have been included in SAPS and APACHE as they have been updated over the years (from three in SAPS II to seven in SAPS III, and from five in APACHE II to seven in APACHE IV), yet still only a small number of comorbidities are included. The predictive value of comorbidity categories as defined by Charlson and colleagues 6 (ten diagnoses) and Elixhauser and colleagues 7 (30 diagnoses) has been studied repeatedly in relation to prediction of mortality in the ICU. In several studies, [8][9][10] Charlson comorbidity categories had better predictive power than the APACHE II comorbidity categories and similar predictive power to the SAPS II, SAPS III, and APACHE II scores. [8][9][10] In another study, 11 comorbidity categories did not contribute significantly to the discrimination of the APACHE II score, and replacement of APACHE II comorbidities with Charlson or Elixhauser comorbidities did not improve discriminatory ability. Two studies 12,13 done in the same homogeneous population of predominantly male, American, military veteran, medical ICU patients showed that the Elixhauser comorbidity categories outperformed APACHE II categories and had better ability to predict ICU mortality because of the inclusion of more than 5000 unique diagnosis codes. Several disease-spectrum-wide studies [14][15][16] have shown that factors such as total burden of disease, time between diagnoses, and order of diagnoses affect the risk of future complications. However, because long-term disease histories are often not systematically recorded everywhere, integration of such information into clinical decision support is still largely unexplored.
In this study, we used neural networks to combine longterm disease histories before ICU admission for more than 230 000 patients from a population-wide disease registry with acute physiology measures obtained from electronic patient records (EPRs) during the first 24 h of the ICU stay to predict in-hospital, 30-day, and 90-day mortality.

Data sources
In this study, we used registry data for long-term disease history and ICU admission, and acute ICU clinical data

Research in context
Evidence before this study Intensive-care units (ICUs) treat highly heterogeneous patients at high risk of mortality. The heterogeneity of the patient population complicates treatment. Linear models that predict patient outcome on the basis of acute physiology measurements are used in clinical practice to support decision making. We based our search for ICU mortality prediction models on a comprehensive 2016 review by Johnson and colleagues. We searched Google Scholar for studies citing the Medical Information Mart for Intensive Care database, the largest online ICU database, that were published in English between Dec 1, 2011 (the date of the first Medical Information Mart for Intensive Care publication), and Aug 1, 2018. In many studies published in the past 10 years, advanced modelling and machine learning techniques, some of which included text-mined features from electronic patient records, have shown promise for improving prediction of prognosis in the ICU based on acute physiology. However, we identified no studies or available severity scoring systems (eg, the Simplified Acute Physiology Score, the Acute Physiologic Assessment and Chronic Health Evaluation) in which long-term disease history before ICU admission was used to predict ICU mortality, even though ICU patients often have long disease histories. In the past 5 years, we have done several disease-spectrum-wide studies showing that factors such as total disease burden, time between diagnoses, and the sequential order of diagnoses affect the risk of future complications. Therefore, the aggregation of long-term disease history and acute physiology measures in a machine learning model could potentially provide a more accurate outcome prediction model to support decision making in the ICU.

Added value of this study
Long-term disease history is seldom systematically exploited in clinical settings, and is rarely used to inform clinical decision making. To our knowledge, our study is the first in which machine learning was used to predict mortality for ICU patients on the basis of long-term disease history. We used disease histories from more than 230 000 ICU patients with up to 23 years of available data before ICU admission stored in a national disease registry. These data were aggregated with acute physiology measurements obtained from electronic patient records. Models based on previous disease history could predict mortality as well as the clinical severity scores in use. Aggregation of long-term disease history and acute physiology measures in a neural network showed that health-related events from different timepoints in a patient's life interact in a non-linear manner, and that taking account of these interactions between long-term and short-term disease history gave more precise prognostic estimates than either long-term disease history or short-term disease history individually.

Implications of all the available evidence
Our study shows the importance of previous disease history in predictions of mortality in ICU patients and thus, the importance of these data in clinical decision making. The predictive value of long-term disease history was stable over time compared with that of physiology measures, is independent of ICU care, and can be made available at admission to the ICU.
from EPRs, to create a machine learning model to predict 30-day and 90-day mortality. Long-term disease histories for all medical and surgical patients admitted to an ICU in Denmark between Jan 1, 2004 (when systematic registration of ICU admissions began), and July 1, 2016, were extracted from the Danish National Patient Registry. 17 The Danish National Patient Registry contains disease histories coded according to the 10th revision of the International Classification of Diseases (ICD-10) dating back to the introduction of ICD-10 terminology in Denmark in 1994. ICD-10 terminology is organised into levels: 21 chapters, 227 diagnosis blocks, and 1698 level-3 diagnoses with more detailed descriptions. The numbers of diagnosis blocks and level-3 diagnoses can vary slightly between countries.
Registry data were excluded if they related to patients younger than 18 years or if outcome information was unavailable because of migration, change of personal identification number, or other similar reasons.
Additionally, raw data for acute ICU measures were extracted from EPRs covering individual medical and surgical patients at three ICUs in hospitals in the Capital Region of Denmark, from which harmonised, highfrequency data were available. Data were extracted from the Daintel Critical Information System, a specialised, commercial data collection and EPR system for ICUs.
After we developed our model, we obtained an external validation dataset from a fourth hospital in the Capital Region of Denmark between June 7, 2012, and May 20, 2016. These data were processed in the same way as the EPR data included in our model. The study was approved by the Danish Patient Safety Authority

Procedures
Dates of admission to, and discharge from, hospital before and after ICU admission were also extracted from the registry and used to calculate time from diagnosis to admission, length of hospital stay before ICU admission, and time to outcome (ie, survivor or non-survivor). Patients with multiple ICU admissions were included in the study (all admissions were used in the model). We retrieved information about date of birth, date of death, and sex from the Danish Central Person Registry. On the basis of this information, we calculated in-hospital mortality (which was defined as death on any ward during hospital admission), 30-day mortality (death within 30 days of ICU admission) and 90-day mortality (death within 90 days of ICU admission). The in-hospital mortality was calculated based on how many patients die while still at the hospital, which could be any number of days after admission. The 30-day mortality was measured 30 days after ICU admission, irrespective of whether patients were inpatients or outpatients. The acute ICU measures from EPR data included in this study were those from the original SAPS II and APACHE II scores, which were in clinical use during the study period. 2,3 SAPS II measures include age, type of admission, three chronic disease variables (metastatic cancer, haematological malignancy, and AIDS), and 12 physiological variables all recorded within the first 24 h of ICU admission (ie, heart rate, temperature, systolic blood pressure, partial pressure of arterial oxygen in inspired air, fractional concentration of oxygen in inspired air, urine output, white blood cell count, bilirubin concentration, and serum urea, bicarbonate, sodium, and potassium concentrations). SAPS II scoring is based on physiological extremes, and therefore both minimum and maximum values for physiological variables were included in our model as variables. These categorical physiological variables were represented as individual binary features, resulting in 27 variables in our model. APACHE II also includes physiological, chronic disease, and admission data. We created maximum and minimum features for the additional physiological variables of APACHE II that are not included in SAPS II, and represented additional comorbidity categories in binary form. 44 variables from the two scores were included in our model. We applied the same exclusion criteria used for the registry data to EPR data. ICU stays shorter than 24 h were also excluded, as were patients with missing outcome data at 90 days. We set a cutoff of one missing value per admission. For all EPR variables, possible physiological ranges were defined from clinical experience (appendix) and values outside these ranges were defined as missing. Missing values were imputed with the median and mode from the feature distributions of continuous and binary features, respectively. We tried other methods, including multiple imputation, but use of the median and mode was simpler and was sufficient for this work in which the amount of missing data was very low.
We used a combination of Tukey's and Winsor's methods 18,19 to normalise data for the continuous variables. Values outside the upper and lower bounds of 1⋅5 times the IQR were set to the upper and lower limits of the range (appendix) to reduce the effects of outliers and produce a normal distribution while conserving the topology of the data. Finally, continuous variables were scaled to a mean of 0 with an SD of 1. The external validation dataset was imputed and normalised with mean and SD as described for the dataset used to develop the model. For both imputation and normalisation, the mean and SD values found for the development dataset were used. All patients were assigned a specific letter (A, B, or C), which represented the hospital that they had been admitted to. This information was presented to the model by means of one-hot encoding with three input units. For the data from the fourth hospital (the external test set), this information was omitted. Our external predictive performance estimate was thus fully independent of information about the group of hospitals represented in the training data.

Model development
Neural network models were trained on data that met inclusion criteria from both sources (ie, registry data and EPR data) both separately (registry data) and in combination with a cross-validation scheme (for data present in both registry and EPRs that met all inclusion criteria) in addition to an independent test set (figure 1). Registry data for ICU admissions were randomly split into a training set (85%) and an independent test set (15%). To avoid overfitting, five-fold cross-validation was done during training. Training data were thus divided into training (80%) and validation (20%) sets for each cross-validation fold. With these percentages (by contrast with a ten-fold cross-validation, for example) the independent test and validation sets were reasonably large, which reduced the risk that they might not be representative of the underlying population-a scenario that could have resulted in overestimation of the predictive performance. 20 A feed-forward neural network with one hidden layer was trained by backpropagation on the training set for 3000 epochs. In each epoch, the performance was assessed against the validation set, and the optimal model for each cross-validation fold was selected from the epoch with the highest performance. We used balanced training to ensure optimal prediction of both classes. Balancing was done in each epoch by randomly picking an equal number of patients from the majority (survivors) and minority (non-survivors) classes.
To establish how length of previous disease history affects ICU mortality, neural networks were trained for datasets with disease history of 1 month, 3 months, 6 months, 1 year, 2⋅5 years, 5 years, 7⋅5 years, 10 years, and 23 years before ICU admission. To establish how detailed the data needed to be represented, additional neural networks were trained: 5 years and 10 years accumulated or year-wise. For example, in the accumulated representation, 5 years of chapter-level diagnoses resulted in 21 variables (one for each chapter), whereas the yearwise representation of five years of ICD-10 chapters resulted in 105 variables. To establish the effect of cohort size, neural networks were trained for cohorts of 250, 500, 1000, 2500, 5000, 10 000, 20 000, 40 000, 80 000, 160 000, and 230 000. All models were trained with chapter-level (n=21) and block-level (n=227) ICD-10 diagnosis codes and with 90-day mortality as the outcome. The optimisation function (RMSprop), activation function (sigmoid), number of batches (one), and learning rate (0⋅001) were kept constant across models. The number of hidden neurons was optimised (0, 25, 100, 250, 500, 750, and 1000 units). We also ran grid searches on optimisation function, activation function, learning rates, number of batches, and different drop-out models. However, we concluded that the number of hidden neurons was most important in relation to predictive performance. We did not include balanced bootstrapping because of the additional computational time that this process would necessitate. A single-layer network of the type that we used can approximate any continuous function from a compact interval of the real numbers, and is therefore not limited in terms of mapping input vectors to mortality labels. 21 The choice of a simple network structure also makes it easier to explain how the model works to users.
The subset of patients for whom EPR data were available was also randomly split into a training set (85%) and an independent test set (15%). Training was done by balanced training with five-fold cross-validation. A one-hidden-layer neural network was trained on the training set for 5000 epochs. For each dataset, the mortality modelling was based on five configurations: "history before admission" was based on 10 years of disease history before ICU admission, age, and sex; "history at admission" was based on data available at ICU admission, including 10 years of disease history, length of hospitalisation before ICU admission, transfer category (ie, medical, scheduled surgery, or unscheduled surgery), and hospital code; "SAPS II" was based on the original SAPS II score; "APACHE II" was based on the original APACHE II score; and "aggregated history" was based on the features from SAPS II, APACHE II, and "history at admission" (appendix). All models were trained for inhospital, 30-day, and 90-day mortality.
Optimisation of hyperparameters was done for 0 (logistic regression), 3, 5, 10, 25, 50, 100, 200, and 300 hidden neurons. Optimisation function (RMSprop), activation function (sigmoid), number of batches (one) and learning rate (0⋅001) were kept constant across EPR models. The optimal models identified in each crossvalidation fold and for each dataset and outcome were used to predict outcome for admissions in the independent test set. The five predictions, one for each cross-validation fold, were gathered in an ensemble (by mean) into the final prediction. The Python code for training the neural network for "aggregated history" is available in the appendix.

Model interpretation
For the best-performing model (trained on aggregated data), we analysed feature importance and feature interactions. We interpreted overall feature importance with the test set. Importance was assessed by estimating the effect of feature absence-ie, in an iterative process, each feature was set to 0 (mean or mode) and new predictions were computed. For each patient, the effect of feature absence was calculated by obtaining the distance to the original prediction. A negative distance was obtained if the new predicted value was higher (ie, towards non-survival) than the original prediction, suggesting that the patient in question had a value that lowered mortality risk compared with the mean risk for the population. A mean of absolute distances across patients was used to generate a ranked list of variables according to effect on outcome predictions across all patients. Estimation of variables' importance with a more sophisticated approach (ie, local interpretable model-agnostic explanations) produced very similar results. 22 Individual variable interactions were calculated for the top-ranking variables. Again, the variable in question was set to 0 for the entire dataset, and each of the remaining features were iteratively set to 0 to estimate the effect of both features being absent at the same time. The effect of the interaction was compared with the additive effect of the variables individually. The difference between the sum of the effects when removing the two features individually and the effect of removing the two features simultaneously was calculated for each patient and the mean of the absolute distances was used to find the top interacting features. Because many of the binary variables were present in only some patients, importance was re-estimated by correcting for the number of patients in whom the variable was present. Some variables were present in less than 1% of patients (n=15) in the independent test set and were not included in the analysis. Analyses were repeated for each model in the ensemble (each cross-validation fold) and the effect was averaged per patient before calculation of the overall importance and strength of the feature interaction.

Figure 1: Study overview of data and time-course aggregation
Previous disease history and acute physiology data from the first 24 h of ICU admission were used separately or in combination as inputs to a neural network algorithm to predict in-hospital, 30-day, and 90-day mortality. Disease history data were retrieved from a population-wide registry that included up to 23 years of disease history before ICU admission. Laboratory values, vital signs, and admission information for the first 24 h of ICU admission were retrieved from electronic patient records from hospitals in the Danish Capital Region. ICU=intensive-care unit. ICD-10=10th revision of the International Classification of Diseases. Model performance was assessed with the Matthews correlation coefficient, the area under the receiver operating characteristics (AUROC) curve, and positive predictive value. The Matthews correlation coefficientwhich is a quality measure for multiclass classifiers that takes the entire confusion matrix into account and, thus, fairly summarises the prediction on unbalanced datasets-was calculated as in which TP=true positives, TN=true negatives, FP=false positives, and FN=false negatives. AUROC was obtained by plotting the rate of correctly classified positives among all positive predictions (ie, the true positive rate) as a function of incorrect positives among all negatives (ie, the false positive rate), at varying thresholds. Positive predictive value, which is also known as precision and is used in clinical practice to assess the performance of alarm systems (for which a low ratio of false alarms is essential), was calculated as

ICD-10 diagnoses Laboratory values, vitals, and admission information
The positive predictive value decreases with increasing number of false positive results. In this study, "negatives" refers to survivors, and "positives" to non-survivors. All analyses were done in Python (version 2.7), in which neural networks were trained with Keras. Plots were generated in R (version 3.4.0).

Role of the funding source
The funders of the study had no role in study design, data collection, data analysis, data interpretation, or writing of the report. The corresponding author had full access to all the data in the study and had final responsibility for the decision to submit for publication.  (table 1).

Results
Model performance, as indicated by the Matthews correlation coefficient, increased as additional years were included in the model, with saturation at 5 years and 10 years of disease history for the chapter and block level diagnoses, respectively (appendix). At both the chapter level and block level of the ICD-10, representation of the data in a year-wise manner resulted in improved mortality prediction compared with accumulated data representation (appendix). Furthermore, increasingly large cohorts were needed to obtain optimal performance when data were represented with increasing granularity-ie, block-level diagnoses and year-wise representation (appendix). The addition of further years to disease history or inclusion of level-3 diagnoses (ie, diagnosis codes) did not improve the model's performance (data not shown).
The Multimorbidity Index was reported to significantly outperform (ie, to have a greater AUROC than) other multimorbidity scores in the prediction of ICU mortality compared with other multimorbidity scores. 13 We also applied the Multimorbidity Index method to our longterm disease history data to predict mortality risk. 10 years of previous history was used with year-wise representation. Our neural-network approach performed better than the Multimorbidity Index in the independent test set for all performance measures (Matthews correlation coefficient 0⋅265 vs 0⋅065; AUROC 0⋅713 vs 0·554; positive predictive value 0⋅388 vs 0⋅257; appendix).
After first training models solely on previous disease history, we studied the predictive effect of previous   figure 3). AUROC and positive predictive value were also improved in the aggregated history model (table 2). Data from the external validation set were obtained between June 7, 2012, and May 20, 2016, and comprised 1384 medical and surgical patients with 1528 ICU admissions ( figure 2B). The best-performing model, the aggregated history model, was validated in the external dataset. For 90-day mortality, the Matthews correlation coefficient was 0·382, the AUROC was 0·746, and the positive predictive value was 0·576 (table 2).
To assess if our initial predictions were reliable as a reference at later stages during ICU stays, we investigated how accurate our initial predictions were for patients remaining in the ICU. A substantial proportion of patients was discharged within the first 10 days of ICU admission (appendix). Importantly, the performance of the models including previous disease history and admission information did not deteriorate during the entire admission irrespective of length of stay in the ICU, whereas predictions based on data collected within the first 24 h of ICU admission become less reliable as time passed (appendix). For the model based on history at admission, the Matthews correlation coefficient was 0·357 after 24 h and 0·337 after 14 days, whereas the Matthews correlation coefficient for the aggregated history model was 0·410 after 24 h and 0·242 at 14 days.
We next interpreted the machine learning model. 22,23 The ten most important variables contributing to the performance of the aggregated history model with aggregated time-course are listed in figure 4A. Low age decreases mortality risk whereas high age increases mortality risk ( figure 4A). The only binary variables among the top ten were hospital A and mechanical ventilation ( figure 4A).
For example, a previous history of diagnoses related to reproduction (ICD-10 diagnoses Z30-39) generally decreased mortality risk, and this effect was stronger for older patients than for younger patients (figure 4B). We also investigated these diagnoses individually: lowrisk women had previous diagnoses related to pregnancy and deliveries (data not shown). Figure 4C shows the interaction of length of stay before ICU admission and the history of haematological malignancy. Mortality risk increases with increasing length of hospital stay before ICU admission, and this effect is enhanced for patients with a history of haematological malignancies ( figure 4C). Yet, in the case of increased mortality risk caused by both a long length of stay and haematological malignancy, the effects of the two variables are not additive; instead a plateau is reached for patients with long hospital stays (>18 days) before ICU admission. The two largest subgroups of ICU patients were patients with respiratory insufficiency, who accounted for 618 (39·7%) of 1556 test set patients, and patients with sepsis, who accounted for 224 (14·4%) of test set patients. Models based mainly on acute measures (eg, SAPS II, APACHE II) generally performed worse in patients with sepsis than in the overall cohort, whereas the aggregated model and the models based on longterm disease history performed well in patients with sepsis ( figure 5A). Furthermore, long-term disease history seemed to contribute more than acute measures to models for estimation of prognosis in patients with sepsis ( figure 5A). Performance of all models in the respiratory insufficiency subgroup was similar to that in the full ICU population ( figure 5B).

Discussion
In this retrospective study of registry data for all Danish ICU admissions between 2004 and 2016 and EPR data for more than 10 000 patients, prediction models for mortality risk in ICU patients based solely on previous disease history performed as well as mortality scores in clinical use (ie, SAPS II and APACHE II). Furthermore, models based on aggregation of long-term disease history and acute physiology measures in a neural network showed that health-related events from different timepoints in a patient's life interact, and that exploitation of these interactions between long-term and short-term disease history gives more precise prognostic estimates than either short-term or long-term history individually.
On the basis of a review of mortality prediction models that included data from the critical care database Medical Information Mart for Intensive Care, we identified studies that applied various machine learning methods and that had outcome measures and inclusion criteria similar to those used in our study. [23][24][25][26][27][28][29][30] Sufficient information to calculate the Matthews correlation coefficient, AUROC, and positive predictive value was available in only three studies. 25,27,28 Comparison of prediction performance showed that other models had higher AUROC but worse Matthews correlation coefficients and positive predictive values than our model (appendix). The Matthews correlation coefficient and positive predictive value are useful performance measures for imbalanced datasets, because they show how well models perform in the two balanced classes and the minority class, respectively. Thus, our model is associated with fewer false predictions of death than those in previous studies.
The value of longitudinal historical health data in predictions of future health has been questioned. 31 In our study, the accuracy of mortality risk prediction was improved by the inclusion of increasingly long disease histories despite changes in clinical practice over the years. Furthermore, our neural network predicted ICU prognosis more accurately than the best-performing multimorbidity score reported so far, the Multimorbidity Index. 13 In the study of the Multimorbidity Index, 13 Min and colleagues argued that diagnoses should not be assumed to independently affect the odds of mortality. Our findings support this position. We did not have longterm data for medications prescribed, adherence to medication, or polypharmacy. Such data could be relevant to our model.
We could not study model interactions of long-term disease history and 24 h ICU measures in a year-wise manner because of the small size of the ICU cohort with both registry and EPR data available (n=10 368). Yet in the larger, population-wide ICU cohort (n=275 143 ICU admissions), in the model based on previous disease history alone, time-wise separation of an individual's diagnoses improves mortality prediction. Furthermore, mortality risk is affected not only by the diagnoses but also the temporal order of these diagnoses (appendix). A larger cohort with fine-grained EPR data would probably improve model performance. Recent disease history (ie, disease history immediately before the hospital admission) affects a patient's risk of ICU mortality more than diseases diagnosed 5-10 years ago. It seems intuitive that diseases that are diagnosed close to ICU admission affect mortality risk more than diseases diagnosed several years previously, although the inclusion of these older diagnoses still further improves mortality prediction models.  A general limitation of our study was that the reliability of the predictive value of the aggregated history models deteriorates over time. Thus, the physiology measures included in the model should be updated dynamically, and prognosis reassessed on a day-by-day basis. However, one of the key clinical findings of our study was that the mortality predictions with models based on previous disease history alone remained stable and reliable throughout an ICU admission, whereas the accuracy of predictions with models based on ICU data deteriorated more rapidly over time (appendix). Furthermore, mortality prediction models based on previous disease history were independent of the care provided during the ICU admission and thus are an unbiased severity measure. The model can be used in the same way as severity scores like SAPS II. The addition of real-time data from ICU care could make the model dynamic and could, for example, help to provide hourly predictions. Additionally, this type of prediction does not require new data collection because it is based on data from an existing high-quality, population-wide registry. Thus, this model overcomes the burden of data harmonisation and quality control to some extent. 17 Practically, mortality predictions could be made before potential ICU admissions and available immediately, by contrast with the scores used in ICUs at present.
Clinical intervention trials do not use the methods that we used in this study, which allowed for initial stratification based on a more accurate understanding of disease history. Patterns in disease history data could probably be used effectively in trial design to detect a difference between groups and to avoid use of noncomparable subgroups. 32 Furthermore, the effect of previous disease history-both diagnosis and the temporal order of diagnoses-on patient outcomes could reflect a transition of physiological systems that predisposes individuals not only to certain diagnoses but also to certain outcomes. This could explain why some survivors of sepsis return to their baseline performance status whereas others progressively decline (a decline not predicted by hospital course). 33   The history before admission model was trained on age, sex, and 10-year disease history before ICU admission. The history at admission model also included transfer category (ie, medical, scheduled surgery, or unscheduled surgery), length of stay before ICU admission, and hospital code. The aggregated history model included an aggregated dataset with both long-term disease history (ie, the data used in the history at admission model) and data for the first 24 h of ICU admission (ie, features included in SAPS II and APACHE II combined). In these three models, disease history was presented as an accumulated representation because the cohort was too small for year-wise representation (appendix). In figure 3B Numerous studies have shown the rapid changes that occur in clinical practice when physicians are offered tools that enable data-guided decision making. 36 However, a clinical trial is required to fully uncover the value of mortality prediction as a decision-support tool in ICUs. We specifically used neural networks because we wanted to study the interaction of health-related events from different timescales. We compared the performance of our aggregated history model with other ICU mortality prediction models. Yet, as highlighted by Johnson and  colleagues, 23 comparison of studies is difficult because of variation in outcome measures, inclusion criteria, feature availability, and time from data collection to mortality prediction. Additionally, AUROC was the only performance metric reported consistently across studies, and this metric does not always adequately reflect model performance for data with unbalanced classes. 37 We found that studies with higher class imbalance (ie, few non-survivors compared with survivors) had better AUROC results but worse Matthews correlation coefficients and positive predictive values than studies with lower class imbalance. That is, high AUROC results can be at the expense of more false positive results, which are penalised more heavily in the calculation of the Matthews correlation coefficient and positive predictive value. Class imbalance was accounted for during training of our model, because there is clinical demand for accurate prediction, especially among nonsurvivors (ie, the minority class). Minimisation of the number of false positive predictions-ie, survivors predicted not to survive-was imperative because such predictions can have fatal consequences if, for example, treatment is stopped on the basis of prediction of a high risk of mortality. Even though our model was validated against external, retrospective data, validation in a prospective study is also important. Neural networks can predict mortality in patients in the ICU as well as or better than other machine learning techniques. 29,38,39 However, interpretation of the feature interaction effects modelled here has not been previously reported. The advantage of the simple approach that we used was higher transparency and much faster computational time. Our study shows that more realistic interpretations of the risk of death can be achieved with neural network models than by studying the additive effects of disease history and ICU measures. Binary variables make a constant contribution to predictions in linear models (eg, SAPS II, APACHE II), but in the neural network their effect varies because of interactions with other features. Because binary features were mostly absent from patients in the EPR cohort, continuous features dominate the list of features whose effect is strongest on mortality across all patients. Explainable neural networks are also gaining popularity in the context of deep learning. 22,40 We believe that explainable models are a key to the transformation of advanced models into actionable, transparent, and trustworthy clinical tools.

Contributors
AP and SB conceived the study, which was designed by ABN. ABN and H-CT-M. did the data analysis, which was interpreted by ABN, H-CT-M, KB, APN, CET, ML, PLM, AP and SB. PJC, MH, LD, LS, and PH extracted and handled data. All authors contributed to the preparation of the Article and approved the final version.

Declaration of interests
LD, LS, and PH are employed by Daintel (which is taking part in the BigTempHealth project funded by the Danish Innovation Fund). AP reports grants from Ferring and the Novo Nordisk Foundation. SB reports personal fees from Intomics and Proscion. All other authors declare no competing interests.

Data sharing
Data are available for use in secure, dedicated environments via application to the Danish Patient Safety Authority and the Danish Health Data Authority.

Figure 5: Mortality prediction for patients in intensive-care units with sepsis (A) and respiratory insufficiency (B) as primary diagnoses
In patients with sepsis, the models built on long-term disease history were better than those built primarily on acute measures. In patients with respiratory insufficiency, performance of all models was similar to that in the full population. SAPS II=Simplified Acute Physiology Score II. APACHE II=Acute Physiologic Assessment and Chronic Health Evaluation II.