Local Calibration: Metrics and Recalibration
  Probabilistic classifiers output confidence scores along with their
predictions, and these confidence scores should be calibrated, i.e., they
should reflect the reliability of the prediction. Confidence scores that
minimize standard metrics such as the expected calibration error (ECE)
accurately measure the reliability on average across the entire population.
However, it is in general impossible to measure the reliability of an
individual prediction. In this work, we propose the local calibration error
(LCE) to span the gap between average and individual reliability. For each
individual prediction, the LCE measures the average reliability of a set of
similar predictions, where similarity is quantified by a kernel function on a
pretrained feature space and by a binning scheme over predicted model
confidences. We show theoretically that the LCE can be estimated
sample-efficiently from data, and empirically find that it reveals
miscalibration modes that are more fine-grained than the ECE can detect. Our
key result is a novel local recalibration method LoRe, to improve confidence
scores for individual predictions and decrease the LCE. Experimentally, we show
that our recalibration method produces more accurate confidence scores, which
improves downstream fairness and decision making on classification tasks with
both image and tabular data.
