{'fraction_non_alphanumeric': 0.02404839185600472, 'fraction_numerical': 0.007229271171437002, 'mean_word_length': 4.794017094017094, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests

Many decisions in medicine involve trade-offs, such as between diagnosing patients with disease versus unnecessary additional testing for those who are healthy. Net benefit is an increasingly reported decision analytic measure that puts benefits and harms on the same scale. This is achieved by specifying an exchange rate, a clinical judgment of the relative value of benefits (such as detecting a cancer) and harms (such as unnecessary biopsy) associated with models, markers, and tests. The exchange rate can be derived by asking simple questions, such as the maximum number of patients a doctor would recommend for biopsy to find one cancer. As the answers to these sorts of questions are subjective, it is possible to plot net benefit for a range of reasonable exchange rates in a “decision curve.” For clinical prediction models, the exchange rate is related to the probability threshold to determine whether a patient is classified as being positive or negative for a disease. Net benefit is useful for determining whether basing clinical decisions on a model, marker, or test would do more good than harm. This is in contrast to traditional measures such as sensitivity, specificity, or area under the curve, which are statistical abstractions not directly informative about clinical value. Recent years have seen an increase in practical applications of net benefit analysis to research data. This is a welcome development, since decision analytic techniques are of particular value when the purpose of a model, marker, or test is to help doctors make better clinical decisions.


Examples of the use of decision curves in the medical literature
There are numerous other examples in the literature where decision curve analysis gives a clear answer as to the clinical value of a marker, test or model, where discrimination and calibration are ambiguous. Here we will focus on four cases for illustrative purposes.
Nam et al. evaluated two prediction models for aggressive prostate cancer, the same scenario discussed in the main text of this paper [1]. The "Sunnybrook" prediction model had an AUC of 0.72 compared to 0.67 for the "Prostate Cancer Prevention Trial" model. The Sunnybrook model overestimated risk, although only slightly at lower risks; the Prostate Cancer Prevention Trial model underestimated lower risks and overestimated higher risks. As might be expected, the decision curve shows that of the two models, the Sunnybrook model has higher net benefit. What is not predictable from the AUCs and calibration plots is that neither model has higher net benefit than the strategy of biopsying all men at risk Predicted Risk unless threshold probabilities are rather high, well above 10%. Because many doctors and patients would consider biopsy at a risk of aggressive prostate cancer less than 10%, the decision curve does not support clinical use of either model.
Another example from cancer screening is an evaluation of a lung cancer prediction tool [2]. The tool has been proposed to determine which patients should be considered for lung CT screening and was evaluated in three cohorts, although for the sake of simplicity, we will focus on just one, the Harvard cohort. The prediction model had an AUC of 0.76, which is reasonable, although not much better than the AUC of 0.74 for smoking duration. Calibration was problematic, with about 30% more events occurring than predicted by the model. The clinical value of a model with modest discrimination and some miscalibration might well be questionable. However, the decision curve clearly demonstrated that across the complete range of reasonable threshold probabilities, the model had higher net benefit than smoking duration alone, or recommending all patients for screening. The authors concluded that the study gave evidence of "benefits for stratifying patients for lung cancer CT screening." A study on a model to predict six-month mortality in elderly patients provides an example where a net benefit approach can come to different conclusions about the value of a model depending on its clinical role [3]. The mortality prediction model was found to have higher net benefit than both assuming all patients die within six months and assuming no patient dies within six months across risk thresholds 5% -25%. The authors concluded that while the model could be of value for determining which patients should be counseled about advance care planning, it should not be used for referral to hospice care. This is on the grounds that whereas a discussion about advance care is indicated for patients with even a relatively low probability of mortality, a hospice referral will generally require more than a 25% risk of death within six months.
Lughezzani et al. provide an example where a decision curve helped choose between two models, one with better discrimination, the other with better calibration [4]. The paper also provides an example where net benefit is used to compare a binary decision rule with a prediction model.
Two high-profile editorials on decision curve analysis have been published in JAMA [5] and the Annals of Internal Medicine [6].

Uncertainty of net benefit estimates
One concern is whether estimates of net benefit should include a measure of uncertainty, such as a 95% confidence interval (C.I.). Although bootstrap resampling methods have been published to calculate a 95% C.I. for net benefit [7], these are not widely used. This may be because uncertainty is a problematic concept in decision theory. An argument that is common among decision analysts is that when we are forced to make a choice between limited options -such as biopsying vs. not biopsying vs. measuring a marker and then deciding -we should choose based on our best guess as to the right choice, irrespective of uncertainty [8]. If we predict that outcome will be best if we biopsy, it is irrelevant whether the chance that we are wrong is 1% or 49%. Others might argue that if we are recommending some change in clinical practice, we ought to be pretty sure that it will do more good than harm.
The table shows one approach to incorporating 95% C.I. into net benefit approaches, based on the data in figure 1 for a study with 1000 patients. At some lower threshold probabilities, the lower bound of the 95% C.I. does include no net benefit of the model compared to the strategy of biopsying all men at risk. Table.
An example of how statistical uncertainty can be incorporated into decision curve analysis. As the model clearly has better net benefit than the both "biopsy none" and "marker", the table shows the 95% C.I. for the difference in net benefit between the model and "biopsy all".