Predicting with Confidence on Unseen Distributions
  Recent work has shown that the performance of machine learning models can
vary substantially when models are evaluated on data drawn from a distribution
that is close to but different from the training distribution. As a result,
predicting model performance on unseen distributions is an important challenge.
Our work connects techniques from domain adaptation and predictive uncertainty
literature, and allows us to predict model accuracy on challenging unseen
distributions without access to labeled data. In the context of distribution
shift, distributional distances are often used to adapt models and improve
their performance on new domains, however accuracy estimation, or other forms
of predictive uncertainty, are often neglected in these investigations. Through
investigating a wide range of established distributional distances, such as
Frechet distance or Maximum Mean Discrepancy, we determine that they fail to
induce reliable estimates of performance under distribution shift. On the other
hand, we find that the difference of confidences (DoC) of a classifier's
predictions successfully estimates the classifier's performance change over a
variety of shifts. We specifically investigate the distinction between
synthetic and natural distribution shifts and observe that despite its
simplicity DoC consistently outperforms other quantifications of distributional
difference. $DoC$ reduces predictive error by almost half ($46\%$) on several
realistic and challenging distribution shifts, e.g., on the ImageNet-Vid-Robust
and ImageNet-Rendition datasets.
