Memory-based Optimization Methods for Model-Agnostic Meta-Learning
  Recently, model-agnostic meta-learning (MAML) has garnered tremendous
attention. However, the stochastic optimization of MAML is still immature.
Existing algorithms for MAML exploit the "episode" idea by sampling some tasks
and some data points for each sampled task at each iteration for updating the
meta-model. However, they either do not necessarily guarantee convergence with
a constant mini-batch size or require processing a large number of tasks at
every iteration, which is not viable for continual learning or cross-device
federated learning where only a small number of tasks are available
per-iteration or per-round. This paper addresses these issues by (i) proposing
efficient memory-based stochastic algorithms for MAML with the vanishing
convergence error, which only requires sampling a constant number of tasks and
a constant number of data samples per-iteration; (ii) proposing
communication-efficient distributed memory-based MAML algorithms for
personalized federated learning in both the cross-device (with client sampling)
and the cross-silo (without client sampling) settings. The theoretical results
significantly improve the optimization theory for MAML and the empirical
results also corroborate the theory.
