Leveraging Unlabeled Data to Predict Out-of-Distribution Performance
  Real-world machine learning deployments are characterized by mismatches
between the source (training) and target (test) distributions that may cause
performance drops. In this work, we investigate methods for predicting the
target domain accuracy using only labeled source data and unlabeled target
data. We propose Average Thresholded Confidence (ATC), a practical method that
learns a threshold on the model's confidence, predicting accuracy as the
fraction of unlabeled examples for which model confidence exceeds that
threshold. ATC outperforms previous methods across several model architectures,
types of distribution shifts (e.g., due to synthetic corruptions, dataset
reproduction, or novel subpopulations), and datasets (Wilds, ImageNet, Breeds,
CIFAR, and MNIST). In our experiments, ATC estimates target performance
$2$-$4\times$ more accurately than prior methods. We also explore the
theoretical foundations of the problem, proving that, in general, identifying
the accuracy is just as hard as identifying the optimal predictor and thus, the
efficacy of any method rests upon (perhaps unstated) assumptions on the nature
of the shift. Finally, analyzing our method on some toy distributions, we
provide insights concerning when it works.
