Improving Classifier Confidence using Lossy Label-Invariant
  Transformations
  Providing reliable model uncertainty estimates is imperative to enabling
robust decision making by autonomous agents and humans alike. While recently
there have been significant advances in confidence calibration for trained
models, examples with poor calibration persist in most calibrated models.
Consequently, multiple techniques have been proposed that leverage
label-invariant transformations of the input (i.e., an input manifold) to
improve worst-case confidence calibration. However, manifold-based confidence
calibration techniques generally do not scale and/or require expensive
retraining when applied to models with large input spaces (e.g., ImageNet). In
this paper, we present the recursive lossy label-invariant calibration (ReCal)
technique that leverages label-invariant transformations of the input that
induce a loss of discriminatory information to recursively group (and
calibrate) inputs - without requiring model retraining. We show that ReCal
outperforms other calibration methods on multiple datasets, especially, on
large-scale datasets such as ImageNet.
