When MAML Can Adapt Fast and How to Assist When It Cannot
  Model-Agnostic Meta-Learning (MAML) and its variants have achieved success in
meta-learning tasks on many datasets and settings. On the other hand, we have
just started to understand and analyze how they are able to adapt fast to new
tasks. For example, one popular hypothesis is that the algorithms learn good
representations for transfer, as in multi-task learning. In this work, we
contribute by providing a series of empirical and theoretical studies, and
discover several interesting yet previously unknown properties of the
algorithm. We find MAML adapts better with a deep architecture even if the
tasks need only a shallow one (and thus, no representation learning is needed).
While echoing previous findings by others that the bottom layers in deep
architectures enable representation learning, we also find that upper layers
enable fast adaptation by being meta-learned to perform adaptive gradient
update when generalizing to new tasks. Motivated by these findings, we study
several meta-optimization approaches and propose a new one for learning to
optimize adaptively. Those approaches attain stronger performance in
meta-learning both shallower and deeper architectures than MAML.
