MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution
  Shifts and Training Conflicts
  Understanding the performance of machine learning models across diverse data
distributions is critically important for reliable applications. Motivated by
this, there is a growing focus on curating benchmark datasets that capture
distribution shifts. While valuable, the existing benchmarks are limited in
that many of them only contain a small number of shifts and they lack
systematic annotation about what is different across different shifts. We
present MetaShift--a collection of 12,868 sets of natural images across 410
classes--to address this challenge. We leverage the natural heterogeneity of
Visual Genome and its annotations to construct MetaShift. The key construction
idea is to cluster images using its metadata, which provides context for each
image (e.g. "cats with cars" or "cats in bathroom") that represent distinct
data distributions. MetaShift has two important benefits: first, it contains
orders of magnitude more natural data shifts than previously available. Second,
it provides explicit explanations of what is unique about each of its data sets
and a distance score that measures the amount of distribution shift between any
two of its data sets. We demonstrate the utility of MetaShift in benchmarking
several recent proposals for training models to be robust to data shifts. We
find that the simple empirical risk minimization performs the best when shifts
are moderate and no method had a systematic advantage for large shifts. We also
show how MetaShift can help to visualize conflicts between data subsets during
model training.
