On Calibration and Out-of-domain Generalization
  Out-of-domain (OOD) generalization is a significant challenge for machine
learning models. Many techniques have been proposed to overcome this challenge,
often focused on learning models with certain invariance properties. In this
work, we draw a link between OOD performance and model calibration, arguing
that calibration across multiple domains can be viewed as a special case of an
invariant representation leading to better OOD generalization. Specifically, we
show that under certain conditions, models which achieve \emph{multi-domain
calibration} are provably free of spurious correlations. This leads us to
propose multi-domain calibration as a measurable and trainable surrogate for
the OOD performance of a classifier. We therefore introduce methods that are
easy to apply and allow practitioners to improve multi-domain calibration by
training or modifying an existing model, leading to better performance on
unseen domains. Using four datasets from the recently proposed WILDS OOD
benchmark, as well as the Colored MNIST dataset, we demonstrate that training
or tuning models so they are calibrated across multiple domains leads to
significantly improved performance on unseen test domains. We believe this
intriguing connection between calibration and OOD generalization is promising
from both a practical and theoretical point of view.
