Should Ensemble Members Be Calibrated?
  Underlying the use of statistical approaches for a wide range of applications
is the assumption that the probabilities obtained from a statistical model are
representative of the "true" probability that event, or outcome, will occur.
Unfortunately, for modern deep neural networks this is not the case, they are
often observed to be poorly calibrated. Additionally, these deep learning
approaches make use of large numbers of model parameters, motivating the use of
Bayesian, or ensemble approximation, approaches to handle issues with parameter
estimation. This paper explores the application of calibration schemes to deep
ensembles from both a theoretical perspective and empirically on a standard
image classification task, CIFAR-100. The underlying theoretical requirements
for calibration, and associated calibration criteria, are first described. It
is shown that well calibrated ensemble members will not necessarily yield a
well calibrated ensemble prediction, and if the ensemble prediction is well
calibrated its performance cannot exceed that of the average performance of the
calibrated ensemble members. On CIFAR-100 the impact of calibration for
ensemble prediction, and associated calibration is evaluated. Additionally the
situation where multiple different topologies are combined together is
discussed.
