Meta-Learning Requires Meta-Augmentation
  Meta-learning algorithms aim to learn two components: a model that predicts
targets for a task, and a base learner that quickly updates that model when
given examples from a new task. This additional level of learning can be
powerful, but it also creates another potential source for overfitting, since
we can now overfit in either the model or the base learner. We describe both of
these forms of metalearning overfitting, and demonstrate that they appear
experimentally in common meta-learning benchmarks. We then use an
information-theoretic framework to discuss meta-augmentation, a way to add
randomness that discourages the base learner and model from learning trivial
solutions that do not generalize to new tasks. We demonstrate that
meta-augmentation produces large complementary benefits to recently proposed
meta-regularization techniques.
