Curriculum in Gradient-Based Meta-Reinforcement Learning
  Gradient-based meta-learners such as Model-Agnostic Meta-Learning (MAML) have
shown strong few-shot performance in supervised and reinforcement learning
settings. However, specifically in the case of meta-reinforcement learning
(meta-RL), we can show that gradient-based meta-learners are sensitive to task
distributions. With the wrong curriculum, agents suffer the effects of
meta-overfitting, shallow adaptation, and adaptation instability. In this work,
we begin by highlighting intriguing failure cases of gradient-based meta-RL and
show that task distributions can wildly affect algorithmic outputs, stability,
and performance. To address this problem, we leverage insights from recent
literature on domain randomization and propose meta Active Domain Randomization
(meta-ADR), which learns a curriculum of tasks for gradient-based meta-RL in a
similar as ADR does for sim2real transfer. We show that this approach induces
more stable policies on a variety of simulated locomotion and navigation tasks.
We assess in- and out-of-distribution generalization and find that the learned
task distributions, even in an unstructured task space, greatly improve the
adaptation performance of MAML. Finally, we motivate the need for better
benchmarking in meta-RL that prioritizes \textit{generalization} over
single-task adaption performance.
