Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot
  Classification Benchmark
  Meta and transfer learning are two successful families of approaches to
few-shot learning. Despite highly related goals, state-of-the-art advances in
each family are measured largely in isolation of each other. As a result of
diverging evaluation norms, a direct or thorough comparison of different
approaches is challenging. To bridge this gap, we perform a cross-family study
of the best transfer and meta learners on both a large-scale meta-learning
benchmark (Meta-Dataset, MD), and a transfer learning benchmark (Visual Task
Adaptation Benchmark, VTAB). We find that, on average, large-scale transfer
methods (Big Transfer, BiT) outperform competing approaches on MD, even when
trained only on ImageNet. In contrast, meta-learning approaches struggle to
compete on VTAB when trained and validated on MD. However, BiT is not without
limitations, and pushing for scale does not improve performance on highly
out-of-distribution MD tasks. In performing this study, we reveal a number of
discrepancies in evaluation norms and study some of these in light of the
performance gap. We hope that this work facilitates sharing of insights from
each community, and accelerates progress on few-shot learning.
