Distance-Based Learning from Errors for Confidence Calibration
  Deep neural networks (DNNs) are poorly calibrated when trained in
conventional ways. To improve confidence calibration of DNNs, we propose a
novel training method, distance-based learning from errors (DBLE). DBLE bases
its confidence estimation on distances in the representation space. In DBLE, we
first adapt prototypical learning to train classification models. It yields a
representation space where the distance between a test sample and its ground
truth class center can calibrate the model's classification performance. At
inference, however, these distances are not available due to the lack of ground
truth labels. To circumvent this by inferring the distance for every test
sample, we propose to train a confidence model jointly with the classification
model. We integrate this into training by merely learning from mis-classified
training samples, which we show to be highly beneficial for effective learning.
On multiple datasets and DNN architectures, we demonstrate that DBLE
outperforms alternative single-model confidence calibration approaches. DBLE
also achieves comparable performance with computationally-expensive ensemble
approaches with lower computational cost and lower number of parameters.
