Net benefit, calibration, threshold selection, and training objectives
  for algorithmic fairness in healthcare
  A growing body of work uses the paradigm of algorithmic fairness to frame the
development of techniques to anticipate and proactively mitigate the
introduction or exacerbation of health inequities that may follow from the use
of model-guided decision-making. We evaluate the interplay between measures of
model performance, fairness, and the expected utility of decision-making to
offer practical recommendations for the operationalization of algorithmic
fairness principles for the development and evaluation of predictive models in
healthcare. We conduct an empirical case-study via development of models to
estimate the ten-year risk of atherosclerotic cardiovascular disease to inform
statin initiation in accordance with clinical practice guidelines. We
demonstrate that approaches that incorporate fairness considerations into the
model training objective typically do not improve model performance or confer
greater net benefit for any of the studied patient populations compared to the
use of standard learning paradigms followed by threshold selection concordant
with patient preferences, evidence of intervention effectiveness, and model
calibration. These results hold when the measured outcomes are not subject to
differential measurement error across patient populations and threshold
selection is unconstrained, regardless of whether differences in model
performance metrics, such as in true and false positive error rates, are
present. In closing, we argue for focusing model development efforts on
developing calibrated models that predict outcomes well for all patient
populations while emphasizing that such efforts are complementary to
transparent reporting, participatory design, and reasoning about the impact of
model-informed interventions in context.
