Learn then Test: Calibrating Predictive Algorithms to Achieve Risk
  Control
  We introduce a framework for calibrating machine learning models so that
their predictions satisfy explicit, finite-sample statistical guarantees. Our
calibration algorithm works with any underlying model and (unknown)
data-generating distribution and does not require model refitting. The
framework addresses, among other examples, false discovery rate control in
multi-label classification, intersection-over-union control in instance
segmentation, and the simultaneous control of the type-1 error of outlier
detection and confidence set coverage in classification or regression. Our main
insight is to reframe the risk-control problem as multiple hypothesis testing,
enabling techniques and mathematical arguments different from those in the
previous literature. We use our framework to provide new calibration methods
for several core machine learning tasks with detailed worked examples in
computer vision and tabular medical data.
