Mitigating Bias in Calibration Error Estimation
  For an AI system to be reliable, the confidence it expresses in its decisions
must match its accuracy. To assess the degree of match, examples are typically
binned by confidence and the per-bin mean confidence and accuracy are compared.
Most research in calibration focuses on techniques to reduce this empirical
measure of calibration error, ECE_bin. We instead focus on assessing
statistical bias in this empirical measure, and we identify better estimators.
We propose a framework through which we can compute the bias of a particular
estimator for an evaluation data set of a given size. The framework involves
synthesizing model outputs that have the same statistics as common neural
architectures on popular data sets. We find that binning-based estimators with
bins of equal mass (number of instances) have lower bias than estimators with
bins of equal width. Our results indicate two reliable calibration-error
estimators: the debiased estimator (Brocker, 2012; Ferro and Fricker, 2012) and
a method we propose, ECE_sweep, which uses equal-mass bins and chooses the
number of bins to be as large as possible while preserving monotonicity in the
calibration function. With these estimators, we observe improvements in the
effectiveness of recalibration methods and in the detection of model
miscalibration.
