Meta-learning with negative learning rates
  Deep learning models require a large amount of data to perform well. When
data is scarce for a target task, we can transfer the knowledge gained by
training on similar tasks to quickly learn the target. A successful approach is
meta-learning, or "learning to learn" a distribution of tasks, where "learning"
is represented by an outer loop, and "to learn" by an inner loop of gradient
descent. However, a number of recent empirical studies argue that the inner
loop is unnecessary and more simple models work equally well or even better. We
study the performance of MAML as a function of the learning rate of the inner
loop, where zero learning rate implies that there is no inner loop. Using
random matrix theory and exact solutions of linear models, we calculate an
algebraic expression for the test loss of MAML applied to mixed linear
regression and nonlinear regression with overparameterized models.
Surprisingly, while the optimal learning rate for adaptation is positive, we
find that the optimal learning rate for training is always negative, a setting
that has never been considered before. Therefore, not only does the performance
increase by decreasing the learning rate to zero, as suggested by recent work,
but it can be increased even further by decreasing the learning rate to
negative values. These results help clarify under what circumstances
meta-learning performs best.
