Measuring Robustness to Natural Distribution Shifts in Image
  Classification
  We study how robust current ImageNet models are to distribution shifts
arising from natural variations in datasets. Most research on robustness
focuses on synthetic image perturbations (noise, simulated weather artifacts,
adversarial examples, etc.), which leaves open how robustness on synthetic
distribution shift relates to distribution shift arising in real data. Informed
by an evaluation of 204 ImageNet models in 213 different test conditions, we
find that there is often little to no transfer of robustness from current
synthetic to natural distribution shift. Moreover, most current techniques
provide no robustness to the natural distribution shifts in our testbed. The
main exception is training on larger and more diverse datasets, which in
multiple cases increases robustness, but is still far from closing the
performance gaps. Our results indicate that distribution shifts arising in real
data are currently an open research problem. We provide our testbed and data as
a resource for future work at https://modestyachts.github.io/imagenet-testbed/ .
