Combining Ensembles and Data Augmentation can Harm your Calibration
  Ensemble methods which average over multiple neural network predictions are a
simple approach to improve a model's calibration and robustness. Similarly,
data augmentation techniques, which encode prior information in the form of
invariant feature transformations, are effective for improving calibration and
robustness. In this paper, we show a surprising pathology: combining ensembles
and data augmentation can harm model calibration. This leads to a trade-off in
practice, whereby improved accuracy by combining the two techniques comes at
the expense of calibration. On the other hand, selecting only one of the
techniques ensures good uncertainty estimates at the expense of accuracy. We
investigate this pathology and identify a compounding under-confidence among
methods which marginalize over sets of weights and data augmentation techniques
which soften labels. Finally, we propose a simple correction, achieving the
best of both worlds with significant accuracy and calibration gains over using
only ensembles or data augmentation individually. Applying the correction
produces new state-of-the art in uncertainty calibration across CIFAR-10,
CIFAR-100, and ImageNet.
