Uncertainty Quantification and Deep Ensembles
  Deep Learning methods are known to suffer from calibration issues: they
typically produce over-confident estimates. These problems are exacerbated in
the low data regime. Although the calibration of probabilistic models is well
studied, calibrating extremely over-parametrized models in the low-data regime
presents unique challenges. We show that deep-ensembles do not necessarily lead
to improved calibration properties. In fact, we show that standard ensembling
methods, when used in conjunction with modern techniques such as mixup
regularization, can lead to less calibrated models. This text examines the
interplay between three of the most simple and commonly used approaches to
leverage deep learning when data is scarce: data-augmentation, ensembling, and
post-processing calibration methods. Although standard ensembling techniques
certainly help boost accuracy, we demonstrate that the calibration of deep
ensembles relies on subtle trade-offs. We also find that calibration methods
such as temperature scaling need to be slightly tweaked when used with
deep-ensembles and, crucially, need to be executed after the averaging process.
Our simulations indicate that this simple strategy can halve the Expected
Calibration Error (ECE) on a range of benchmark classification problems
compared to standard deep-ensembles in the low data regime.
