Learning how to approve updates to machine learning algorithms in
  non-stationary settings
  Machine learning algorithms in healthcare have the potential to continually
learn from real-world data generated during healthcare delivery and adapt to
dataset shifts. As such, the FDA is looking to design policies that can
autonomously approve modifications to machine learning algorithms while
maintaining or improving the safety and effectiveness of the deployed models.
However, selecting a fixed approval strategy, a priori, can be difficult
because its performance depends on the stationarity of the data and the quality
of the proposed modifications. To this end, we investigate a
learning-to-approve approach (L2A) that uses accumulating monitoring data to
learn how to approve modifications. L2A defines a family of strategies that
vary in their "optimism''---where more optimistic policies have faster approval
rates---and searches over this family using an exponentially weighted average
forecaster. To control the cumulative risk of the deployed model, we give L2A
the option to abstain from making a prediction and incur some fixed abstention
cost instead. We derive bounds on the average risk of the model deployed by
L2A, assuming the distributional shifts are smooth. In simulation studies and
empirical analyses, L2A tailors the level of optimism for each problem-setting:
It learns to abstain when performance drops are common and approve beneficial
modifications quickly when the distribution is stable.
