How to distribute data across tasks for meta-learning?
  Meta-learning models transfer the knowledge acquired from previous tasks to
quickly learn new ones. They are trained on benchmarks with a fixed number of
data points per task. This number is usually arbitrary and it is unknown how it
affects performance at testing. Since labelling of data is expensive, finding
the optimal allocation of labels across training tasks may reduce costs. Given
a fixed budget of labels, should we use a small number of highly labelled
tasks, or many tasks with few labels each? Should we allocate more labels to
some tasks and less to others? We show that: 1) If tasks are homogeneous, there
is a uniform optimal allocation, whereby all tasks get the same amount of data;
2) At fixed budget, there is a trade-off between number of tasks and number of
data points per task, with a unique solution for the optimum; 3) When trained
separately, harder task should get more data, at the cost of a smaller number
of tasks; 4) When training on a mixture of easy and hard tasks, more data
should be allocated to easy tasks. Interestingly, Neuroscience experiments have
shown that human visual skills also transfer better from easy tasks. We prove
these results mathematically on mixed linear regression, and we show
empirically that the same results hold for few-shot image classification on
CIFAR-FS and mini-ImageNet. Our results provide guidance for allocating labels
across tasks when collecting data for meta-learning.
