{'fraction_non_alphanumeric': 0.0189648033126294, 'fraction_numerical': 0.008281573498964804, 'mean_word_length': 5.29614181438999, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving the Presentation and Understanding of Risk Models

The key concepts (calibration, discrimination, and discordance) important in understanding and comparing risk models are best conveyed graphically. To illustrate this, models predicting death and acute kidney injury in a large cohort of PCI patients differing in the number of predictors included are presented. Calibration plots, often presented in the current literature, present the agreement between predicted and observed risk for deciles of risk. Risk distribution curves present the frequency of different levels of risk. Scatterplots of the risks assigned to individuals by different models show the discordance of the individual risk estimates. Increasing the number of predictors in these models produce increasingly disperse and progressively skewed risk distribution curves. These resemble the lognormal distributions expected when risk predictors interact multiplicatively. These changes in the risk distribution curves correlate with improved measures of discrimination.


INTRODUCTION
Clinicians are frequently challenged to interpret the literature on risk models and risk factors. A major reason is that publications on predictive models have largely focused on numerical presentations of their development and/or comparison. And the experts may disagree on what numerical measures should be used, what they mean, and their clinical significance. 1 Calibration curves, scatterplots that compare predicted and observed risk in each decile of risk, are often presented. Recently the value of graphical presentations of risk distribution curves or predictiveness curves has been emphasized. 2,3 The former is a graph of the frequency of risk versus risk and the latter a graph of risk versus the cumulative frequency of risk. Pepe, Gu, and Morris have written that "Displaying risk distributions is a fundamental step in evaluating the performance of a risk prediction model, a step that is often overlooked in practice." 4 Another graphical presentation, a scatterplot presenting the discordance of individual risk estimates when two models are compared, was introduced by Lemeshow et al. 5 and has recently been rediscovered by Pencina et al. 6 In this paper we use these graphical presentations to characterize models differing in the number of risk factors providing insights not available from numerical presentations.

METHODS
The study cohort for our analysis included patients undergoing Percutaneous Coronary Intervention (PCI) in years 2007 and 2008 in a large regional registry of contemporary PCI. The details of the Blue Cross Blue Shield of Michigan Cardiovascular Consortium registry (BMC02) and of the data collection process have been described elsewhere. 7,8,9,10 Briefly procedural data on all patients undergoing elective and non-elective PCI at the 31 participating hospitals is collected using standardized data collection forms. Baseline data include clinical, demographic, procedural, and angiographic characteristics as well as medications used before, during, and after the procedure, and in-hospital outcome. All data elements have been prospectively defined and the local institutional review board at each institution approved the protocol. The data is collected by a dedicated staff member and forwarded to the coordinating center. Medical records of all patients undergoing coronary artery bypass grafting (CABG), and of patients who died in the hospital are reviewed by auditors from the coordinating center to ensure data accuracy. A further 2% of cases are randomly selected for audit.
Two outcomes were considered: all-cause in-hospital death (death) and acute kidney injury (AKI). AKI is an in-hospital outcome defined as "peak minus baseline creatinine ≥0.5 mg/dL", with peak creatinine measured in-hospital before discharge. Death was less common (1.10%) than AKI (3.40%).
Of the total cohort, 60,654 patients were included in the death model while 47,775 patients were included in the AKI model. Of the 13,209 patients who were not included in the AKI model, 1238 were on dialysis prior to the procedure while 10595 patients were excluded due to absence of serum creatinine before or after the procedure and 1756 were excluded due to absence of body weight.
We developed a series of models for death and AKI strictly for illustrative purposes. Our goal was to show the effect of increasing numbers of predictors on the risk distribution curve and associated measures. Full models (model 5) for death and AKI were developed by ascending stepwise logistic regression. Then a series of nested submodels (models 1, 2, 3, and 4) with increasing numbers of predictors were constructed for death and AKI. Model 1 had only had age and gender. Subsequently additional predictors were added to generate models 2, 3, and 4, with the strongest predictors added first. Predictors were categorical variables, with the exception of baseline creatinine in the AKI models.
The predictors in the death model were limited to those available prior to PCI, while the predictors in the AKI model also included procedural and angiographic data.
Statistical support for this project was constrained by time, so additional analyses and/or alternative graphics could not be generated.

RESULTS
Tables 1 and 2 summarize the 5 models for death and AKI, respectively. As more predictors are included in the models, measures of goodness of fit and discrimination are progressively improved. Cardiogenic shock was the strongest predictor for death and AKI.
For death, although model 1 has a significant Hosmer-Lemshow statistic, the other models are calibrated by this metric. Figure  Numerical tables, calibration plots, and risk distribution curves describe how addition of predictors improves the risk stratification of the population, but not their effects on individuals. This is best appreciated by inspecting a scatterplots of the individual risk estimates from two models. Figures 5 and   6 do this for models 4 and 5 for death and AKI, respectively. These two models were the closest in terms of risk stratification of the population, but clearly give different results at the individual level.
In theory, risk distribution curves generated from multiple risk factors that interact multiplicatively should be lognormal. 12 In figure 7 simulated lognormal risk distribution curves with mean risks of 0.011 and 0.034, the mean risks of death and AKI, respectively, but differing dispersion are presented for comparison to the risk distribution curves of figures 3 and 4. Curves with low dispersion are symmetrical and centered on the mean. Curves with higher dispersion are progressively skewed to the right.

DISCUSSION
We agree with Jaynes, who wrote "The First Commandment of scientific data analysis publication ought to be: 'Thou shalt reveal thy full original data, unmutilated by any processing whatsoever." 13 Before calculating statistics, especially on categorized data, readers should be given an opportunity to visualize the data.
Numerical results do not provide the insight provided by risk distribution curves. The latter present the location, dispersion, and shape of the risk distribution curve. Comparison of risk distributions differing in the number of predictors allows a better understanding of the modeling process.
Nomenclature is confusing as both calibration and discrimination are referred to as accuracy. However Although additional risk factors do not provide improved calibration or accuracy, they do provide improved discrimination. This is measured by the c-statistic or area under the ROC curve, which are measures of the overlap between the risk distribution curves for cases and controls, both of which are fully determined by the population risk distribution. Improved discrimination reflects the dispersion of the risk distribution curve. Narrow risk distribution curves assign patients who will have events and those who won't have events similar probabilities of an outcome. In this instance there must be substantial overlap of the derived risk distribution curves for cases and controls. On the other hand, as additional risk factors are included, broader risk distribution curves result in cases and controls being assigned increasingly different probabilities of an outcome. This is readily appreciated when a composite plot of risk distribution curves generated by models differing in the number of predictors is presented.
Presenting risk distribution curves separately for cases and controls has been advocated. 14 We agree this well depicts the discrimination of the two groups. However we favor the population risk distribution curve as, in theory, the risk distribution curves for cases and controls can be calculated directly from the risk distribution curve for the population. 15 An additional feature apparent from the composite plot is that the distributions also become increasingly asymmetric as additional predictors are included. When a sufficient number of risk factors interact multiplicatively, the expected distribution of risk in the population is lognormal. 12 We suggest that that is the case with this series of models. Narrow lognormal curves are symmetric and resemble normal distributions. But increasing dispersion is associated with increasingly asymmetric lognormal curves. That simulated lognormal curves with the same mean risks as observed for death and AKI appear similar supports the suggestion of lognormality.
But even the addition of risk distribution curves to numerical presentations provides no insight into how individuals are characterized by different models. This is best appreciated from a scatterplot of the calculated risks for individuals derived from two models, an approach introduced by Lemeshow et al.
When models are nested, as is the case here, this discordance may be less than when there are fewer shared predictors. Nonetheless, even when addition of predictors has little or no impact on the dispersion of the risk distribution curve, there can be significant disagreement. When the risk distribution is split into categories, disagreement has been termed reclassification. However when the models are calibrated and their risk distributions are similar, this reclassification has little clinical significance.
The presentation and understanding of risk models would be improved by including risk distribution curves in all publications. Graphical presentation of the data prior to categorization and/or statistical analysis is standard practice in other areas of science and medicine. Risk distribution curves directly depict the dispersion of the risks assigned by a model. That measures of discrimination reflect this dispersion is an important point that has been overlooked because of the lack of graphical presentation.
When more than one model is utilized, comparison of risk distribution curves allow the observer to assess whether they differ in location (demonstrating a difference in calibration) or dispersion (demonstrating a difference in discrimination). Since risk distribution curves are not presented, there has been no attention paid to their shape and the influence of increasing the number of included predictors on their shape. A normal distribution cannot be assumed and the skewing of distributions would be important information for modeling the clinical benefit of adding additional predictors to a model Scatterplots of the predicted risk for individuals from two models, originally proposed by Lemeshow et al. 5 and more recently suggested by Pencina et al., 6 should also be presented in publications. As for risk distribution curves, this should precede categorization and/or statistical analysis.
The clinical goal of model development is to identify risk factors that produce optimal population risk stratification, which means identification of subpopulations that are as large as possible and differ as much as possible from each other and that are calibrated. Intuitively, this should correspond to a risk distribution curve as disperse as possible.