{'fraction_non_alphanumeric': 0.03295772261289503, 'fraction_numerical': 0.019062950097432856, 'mean_word_length': 5.51794588625069, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Development and validation of the ISARIC 4C Deterioration model for adults hospitalised with COVID-19: a prospective cohort study

Summary Background Prognostic models to predict the risk of clinical deterioration in acute COVID-19 cases are urgently required to inform clinical management decisions. Methods We developed and validated a multivariable logistic regression model for in-hospital clinical deterioration (defined as any requirement of ventilatory support or critical care, or death) among consecutively hospitalised adults with highly suspected or confirmed COVID-19 who were prospectively recruited to the International Severe Acute Respiratory and Emerging Infections Consortium Coronavirus Clinical Characterisation Consortium (ISARIC4C) study across 260 hospitals in England, Scotland, and Wales. Candidate predictors that were specified a priori were considered for inclusion in the model on the basis of previous prognostic scores and emerging literature describing routinely measured biomarkers associated with COVID-19 prognosis. We used internal–external cross-validation to evaluate discrimination, calibration, and clinical utility across eight National Health Service (NHS) regions in the development cohort. We further validated the final model in held-out data from an additional NHS region (London). Findings 74 944 participants (recruited between Feb 6 and Aug 26, 2020) were included, of whom 31 924 (43·2%) of 73 948 with available outcomes met the composite clinical deterioration outcome. In internal–external cross-validation in the development cohort of 66 705 participants, the selected model (comprising 11 predictors routinely measured at the point of hospital admission) showed consistent discrimination, calibration, and clinical utility across all eight NHS regions. In held-out data from London (n=8239), the model showed a similarly consistent performance (C-statistic 0·77 [95% CI 0·76 to 0·78]; calibration-in-the-large 0·00 [–0·05 to 0·05]); calibration slope 0·96 [0·91 to 1·01]), and greater net benefit than any other reproducible prognostic model. Interpretation The 4C Deterioration model has strong potential for clinical utility and generalisability to predict clinical deterioration and inform decision making among adults hospitalised with COVID-19. Funding National Institute for Health Research (NIHR), UK Medical Research Council, Wellcome Trust, Department for International Development, Bill & Melinda Gates Foundation, EU Platform for European Preparedness Against (Re-)emerging Epidemics, NIHR Health Protection Research Unit (HPRU) in Emerging and Zoonotic Infections at University of Liverpool, NIHR HPRU in Respiratory Infections at Imperial College London.


Variable selection
We used a logistic regression modelling approach and performed backward elimination of the a priori candidate variables using Akaike information criteria (AIC). For continuous variables, linear and spline terms were grouped together and considered collectively for inclusion/exclusion in the model. Variables that led to a reduction in AIC were retained (equivalent to p=0.157 for predictors with 1 degree of freedom), using the fastbw command in the rms package in R 1 . This process was done separately in each multiply imputed dataset and in each NHS region in the development set. Predictors were required to be retained in >50% of multiply imputed datasets in >50% of development NHS regions in order to enter the final model. We specified this in order to retain a parsimonious set of predictors that had consistent prognostic value across the development NHS regions.

Missing data
We handled missing data (including predictors and outcomes) using multiple imputation with chained equations, assuming missingness at random 2 , using the mice package in R 3 . We included all predictors (including restricted cubic spline transformations) and the outcome in the imputation models to ensure compatibility. We used predictive mean matching for continuous variables, logistic regression imputation for binary data, polytomous regression imputation for unordered categorical data and proportional odds model for ordered categorical data 3 .
We assessed convergence of the imputation models by examining convergence plots and examined the distributions of imputed values to ensure plausibility. Imputation was done separately for each NHS region to preserve potential inter-region heterogeneity. We generated 10 multiply imputed datasets; all primary analyses were performed in each imputed dataset and model and validation parameters were pooled using Rubin's rules 4 .

Validation metrics
C-statistics were calculated using the pROC package in R 5 . Calibration slopes were calculated by fitting a logistic regression model for the outcome, with the model linear predictor as the sole independent variable; the coefficient for the linear predictor provides the calibration slope metric. Calibration-in-the-large was computed by fitting a logistic regression model for the outcome, with the model linear predictor as an offset term; the intercept of the fitted model provides the calibration-in-the-large. Calibration plots were generated by fitting a loess smoother between model predictions and the outcome in stacked multiply imputed datasets. Recalibration to each region was performed by re-estimating the model intercept in the validation sets during each internal-external crossvalidation cycle.

Sample size
We assumed a priori that at least 30% of hospital admission would reach the primary outcome. The most comparable existing tools for risk stratification are the qSOFA, NEWS2, and CURB-65 scores [9][10][11] . qSOFA achieved C-statistics ranging from 0.71-0.78 during external validation 9 . Based on a conservative C-statistic of 0.71, we assumed a Cox-Snell R 2 of 0.24. With 40 candidate predictor parameters (including transformations), the minimum sample size required for new model development was estimated to be 1,290, with 387 events 12,13 .

Supplementary Figure 1: Days to deterioration following admission, stratified by first chronological type of deterioration.
Outcome events are shown as stacked histogram. Total sample size = 74,944 participants with 30,899 deterioration events with known time point shown in plot. Median days to deterioration was 4 (interquartile range 1-9). Noninvasive = non-invasive ventilation; ICU = high dependency or intensive care unit admission; IMV = invasive mechanical ventilation; ECMO = extra-corporeal membrane oxygenation.

Supplementary Figure 3: Pooled calibration across multiple imputed datasets during IECV (a) before and (b) after recalibration to local regions.
Calibration is shown using a loess-smoother across multiply imputed datasets. Rug plots indicate the distributions of predicted risk. Total sample size = 66,705 participants.

(a) (b) Supplementary Figure 4: Decision curve analysis during internal-external cross validation.
Shown for the 4C Deterioration model without recalibration to local region. Net benefit is shown for each candidate model compared to the 'treat all' and 'treat none' approaches. Points score models are recalibrated to the validation data, resulting in optimistic estimates of net benefit for these models. Total sample size = 66,705 participants.

Supplementary Figure 5: Sensitivity, specificity, positive predictive values (PPVs) and negative predictive values (NPVs) of the 4C Deterioration model in the London validation cohort (n= 8,239 participants).
Each metric is shown across the full range of probability thresholds for the model predictions for illustration. The distribution of predictions is shown in the histogram. Positive predictive value at a threshold of 0% and negative predictive value at a threshold of 100% reflect prior probability due to the overall observed risk of deterioration in the validation cohort.

Supplementary Figure 6: 4C Deterioration vs Mortality predictions for London validation cohort (n = 8,239) shown by (a) sex and (b) ethnicity.
Both plots are shown stratified by age in years. 4C Mortality probabilities are calculated from points scores, based on observed mortality risk for each score in the original validation data. Smoothed lines reflects loess fit.

Supplementary Figure 7: Multivariable associations between predictors and outcome using alternative multiple imputation approach as sensitivity analysis.
Total development sample size = 66,705 participants. Black lines and dots indicate point estimates; grey shaded regions and error bars indicate 95% confidence intervals.

Supplementary Figure 8: Internal-external cross validation of model by NHS region using alternative multiple imputation approach as sensitivity analysis.
Pooled estimates are calculated through random-effects meta-analysis. Total sample size = 66,705 participants. Black squares indicate point estimates; bars indicate 95% confidence intervals; diamonds indicate pooled random-effects meta-analysis estimates.

Supplementary Figure 9: Multivariable associations between predictors and outcome using alternative definition of nosocomial infection (>5 days from admission) as sensitivity analysis.
Total development sample size = 66,705 participants. Black lines and dots indicate point estimates; grey shaded regions and error bars indicate 95% confidence intervals.

Laboratory measures
Haemoglobin (

Radiology
Infiltrates on chest radiograph Factor Yes, No All variables were taken from point of hospital admission, or the day of first clinical assessment for COVID-19., *Comorbidities included chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic renal disease, mild to severe liver disease, dementia, chronic neurological disease, connective tissue disease, diabetes mellitus, HIV or AIDS, malignancy and clinician-defined obesity.

Supplementary Table 4: Validation in complete case London data as sensitivity analysis.
Models are shown for prediction of in-hospital clinical deterioration and are sorted by C-statistic. CITL = calibration-in-the-large. 'Original outcome' column indicates original intended outcome for each candidate model during development. CITL and slopes are not shown for points score models since they are not on probability scale.

Score
Original outcome n C-statistic CITL Slope

Supplementary Table 5: Validation in London data stratified by time from admission or first COVID-19 assessment to deterioration as sensitivity analysis.
C-statistics (with 95% confidence intervals) are shown for each model for prediction of in-hospital clinical deterioration, and stratified by time to deterioration as indicated. 'Original outcome' column indicates original intended outcome for each candidate model during development.

Supplementary Table 7: Validation in London data stratified by (a) community (n = 7,771 participants) vs (b) nosocomial infection (n = 468 participants) as sensitivity analysis.
Models are shown for prediction of in-hospital clinical deterioration and are sorted by C-statistic. CITL = calibration-in-the-large. 'Original outcome' column indicates original intended outcome for each candidate model during development. CITL and slopes are not shown for points score models since they are not on probability scale.

(a)
Score

Supplementary Table 9: Validation parameters of prognostic model in London cohort using alternative multiple imputation approach as sensitivity analysis (n = 8,239 participants).
Models are shown for prediction of in-hospital clinical deterioration and are sorted by C-statistic. CITL = calibration-in-the-large. 'Original outcome' column indicates original intended outcome for each candidate model during development. CITL and slopes are not shown for points score models since they are not on probability scale.