Distributionally Robust Learning for Uncertainty Calibration under
  Domain Shift
  We propose a framework for learning calibrated uncertainties under domain
shifts. We consider the case where the source (training) distribution differs
from the target (test) distribution. We detect such domain shifts through the
use of a binary domain classifier and integrate it with the task network and
train them jointly end-to-end. The binary domain classifier yields a density
ratio that reflects the closeness of a target (test) sample to the source
(training) distribution. We employ it to adjust the uncertainty of prediction
in the task network. This idea of using the density ratio is based on the
distributionally robust learning (DRL) framework, which accounts for the domain
shift through adversarial risk minimization. We demonstrate that our method
generates calibrated uncertainties that benefit many downstream tasks, such as
unsupervised domain adaptation (UDA) and semi-supervised learning (SSL). In
these tasks, methods like self-training and FixMatch use uncertainties to
select confident pseudo-labels for re-training. Our experiments show that the
introduction of DRL leads to significant improvements in cross-domain
performance. We also demonstrate that the estimated density ratios show
agreement with the human selection frequencies, suggesting a positive
correlation with a proxy of human perceived uncertainties.
