Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
  In many applications, it is desirable that a classifier not only makes
accurate predictions, but also outputs calibrated posterior probabilities.
However, many existing classifiers, especially deep neural network classifiers,
tend to be uncalibrated. Post-hoc calibration is a technique to recalibrate a
model by learning a calibration map. Existing approaches mostly focus on
constructing calibration maps with low calibration errors, however, this
quality is inadequate for a calibrator being useful. In this paper, we
introduce two constraints that are worth consideration in designing a
calibration map for post-hoc calibration. Then we present Meta-Cal, which is
built from a base calibrator and a ranking model. Under some mild assumptions,
two high-probability bounds are given with respect to these constraints.
Empirical results on CIFAR-10, CIFAR-100 and ImageNet and a range of popular
network architectures show our proposed method significantly outperforms the
current state of the art for post-hoc multi-class classification calibration.
