On the Convergence Theory of Gradient-Based Model-Agnostic Meta-Learning
  Algorithms
  We study the convergence of a class of gradient-based Model-Agnostic
Meta-Learning (MAML) methods and characterize their overall complexity as well
as their best achievable accuracy in terms of gradient norm for nonconvex loss
functions. We start with the MAML method and its first-order approximation
(FO-MAML) and highlight the challenges that emerge in their analysis. By
overcoming these challenges not only we provide the first theoretical
guarantees for MAML and FO-MAML in nonconvex settings, but also we answer some
of the unanswered questions for the implementation of these algorithms
including how to choose their learning rate and the batch size for both tasks
and datasets corresponding to tasks. In particular, we show that MAML can find
an $\epsilon$-first-order stationary point ($\epsilon$-FOSP) for any positive
$\epsilon$ after at most $\mathcal{O}(1/\epsilon^2)$ iterations at the expense
of requiring second-order information. We also show that FO-MAML which ignores
the second-order information required in the update of MAML cannot achieve any
small desired level of accuracy, i.e., FO-MAML cannot find an $\epsilon$-FOSP
for any $\epsilon>0$. We further propose a new variant of the MAML algorithm
called Hessian-free MAML which preserves all theoretical guarantees of MAML,
without requiring access to second-order information.
