Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD
  We propose a new computationally-efficient first-order algorithm for
Model-Agnostic Meta-Learning (MAML). The key enabling technique is to interpret
MAML as a bilevel optimization (BLO) problem and leverage the sign-based
SGD(signSGD) as a lower-level optimizer of BLO. We show that MAML, through the
lens of signSGD-oriented BLO, naturally yields an alternating optimization
scheme that just requires first-order gradients of a learned meta-model. We
term the resulting MAML algorithm Sign-MAML. Compared to the conventional
first-order MAML (FO-MAML) algorithm, Sign-MAML is theoretically-grounded as it
does not impose any assumption on the absence of second-order derivatives
during meta training. In practice, we show that Sign-MAML outperforms FO-MAML
in various few-shot image classification tasks, and compared to MAML, it
achieves a much more graceful tradeoff between classification accuracy and
computation efficiency.
