Task-Robust Model-Agnostic Meta-Learning
  Meta-learning methods have shown an impressive ability to train models that
rapidly learn new tasks. However, these methods only aim to perform well in
expectation over tasks coming from some particular distribution that is
typically equivalent across meta-training and meta-testing, rather than
considering worst-case task performance. In this work we introduce the notion
of "task-robustness" by reformulating the popular Model-Agnostic Meta-Learning
(MAML) objective [Finn et al. 2017] such that the goal is to minimize the
maximum loss over the observed meta-training tasks. The solution to this novel
formulation is task-robust in the sense that it places equal importance on even
the most difficult and/or rare tasks. This also means that it performs well
over all distributions of the observed tasks, making it robust to shifts in the
task distribution between meta-training and meta-testing. We present an
algorithm to solve the proposed min-max problem, and show that it converges to
an $\epsilon$-accurate point at the optimal rate of $\mathcal{O}(1/\epsilon^2)$
in the convex setting and to an $(\epsilon, \delta)$-stationary point at the
rate of $\mathcal{O}(\max\{1/\epsilon^5, 1/\delta^5\})$ in nonconvex settings.
We also provide an upper bound on the new task generalization error that
captures the advantage of minimizing the worst-case task loss, and demonstrate
this advantage in sinusoid regression and image classification experiments.
