Unlabelled Data Improves Bayesian Uncertainty Calibration under
  Covariate Shift
  Modern neural networks have proven to be powerful function approximators,
providing state-of-the-art performance in a multitude of applications. They
however fall short in their ability to quantify confidence in their predictions
- this is crucial in high-stakes applications that involve critical
decision-making. Bayesian neural networks (BNNs) aim at solving this problem by
placing a prior distribution over the network's parameters, thereby inducing a
posterior distribution that encapsulates predictive uncertainty. While existing
variants of BNNs based on Monte Carlo dropout produce reliable (albeit
approximate) uncertainty estimates over in-distribution data, they tend to
exhibit over-confidence in predictions made on target data whose feature
distribution differs from the training data, i.e., the covariate shift setup.
In this paper, we develop an approximate Bayesian inference scheme based on
posterior regularisation, wherein unlabelled target data are used as
"pseudo-labels" of model confidence that are used to regularise the model's
loss on labelled source data. We show that this approach significantly improves
the accuracy of uncertainty quantification on covariate-shifted data sets, with
minimal modification to the underlying model architecture. We demonstrate the
utility of our method in the context of transferring prognostic models of
prostate cancer across globally diverse populations.
