Multi-Class Uncertainty Calibration via Mutual Information
  Maximization-based Binning
  Post-hoc multi-class calibration is a common approach for providing
high-quality confidence estimates of deep neural network predictions. Recent
work has shown that widely used scaling methods underestimate their calibration
error, while alternative Histogram Binning (HB) methods often fail to preserve
classification accuracy. When classes have small prior probabilities, HB also
faces the issue of severe sample-inefficiency after the conversion into K
one-vs-rest class-wise calibration problems. The goal of this paper is to
resolve the identified issues of HB in order to provide calibrated confidence
estimates using only a small holdout calibration dataset for bin optimization
while preserving multi-class ranking accuracy. From an information-theoretic
perspective, we derive the I-Max concept for binning, which maximizes the
mutual information between labels and quantized logits. This concept mitigates
potential loss in ranking performance due to lossy quantization, and by
disentangling the optimization of bin edges and representatives allows
simultaneous improvement of ranking and calibration performance. To improve the
sample efficiency and estimates from a small calibration set, we propose a
shared class-wise (sCW) calibration strategy, sharing one calibrator among
similar classes (e.g., with similar class priors) so that the training sets of
their class-wise calibration problems can be merged to train the single
calibrator. The combination of sCW and I-Max binning outperforms the state of
the art calibration methods on various evaluation metrics across different
benchmark datasets and models, using a small calibration set (e.g., 1k samples
for ImageNet).
