Optimal sizing of a holdout set for safe predictive model updating
  Risk models in medical statistics and healthcare machine learning are
increasingly used to guide clinical or other interventions. Should a model be
updated after a guided intervention, it may lead to its own failure at making
accurate predictions. The use of a `holdout set' -- a subset of the population
that does not receive interventions guided by the model -- has been proposed to
prevent this. Since patients in the holdout set do not benefit from risk
predictions, the chosen size must trade off maximising model performance whilst
minimising the number of held out patients. By defining a general loss
function, we prove the existence and uniqueness of an optimal holdout set size,
and introduce parametric and semi-parametric algorithms for its estimation. We
demonstrate their use on a recent risk score for pre-eclampsia. Based on these
results, we argue that a holdout set is a safe, viable and easily implemented
solution to the model update problem.
