The Effect of Diversity in Meta-Learning
  Few-shot learning aims to learn representations that can tackle novel tasks
given a small number of examples. Recent studies show that task distribution
plays a vital role in the model's performance. Conventional wisdom is that task
diversity should improve the performance of meta-learning. In this work, we
find evidence to the contrary; we study different task distributions on a
myriad of models and datasets to evaluate the effect of task diversity on
meta-learning algorithms. For this experiment, we train on multiple datasets,
and with three broad classes of meta-learning models - Metric-based (i.e.,
Protonet, Matching Networks), Optimization-based (i.e., MAML, Reptile, and
MetaOptNet), and Bayesian meta-learning models (i.e., CNAPs). Our experiments
demonstrate that the effect of task diversity on all these algorithms follows a
similar trend, and task diversity does not seem to offer any benefits to the
learning of the model. Furthermore, we also demonstrate that even a handful of
tasks, repeated over multiple batches, would be sufficient to achieve a
performance similar to uniform sampling and draws into question the need for
additional tasks to create better models.
