Unsupervised Curricula for Visual Meta-Reinforcement Learning
  In principle, meta-reinforcement learning algorithms leverage experience
across many tasks to learn fast reinforcement learning (RL) strategies that
transfer to similar tasks. However, current meta-RL approaches rely on
manually-defined distributions of training tasks, and hand-crafting these task
distributions can be challenging and time-consuming. Can "useful" pre-training
tasks be discovered in an unsupervised manner? We develop an unsupervised
algorithm for inducing an adaptive meta-training task distribution, i.e. an
automatic curriculum, by modeling unsupervised interaction in a visual
environment. The task distribution is scaffolded by a parametric density model
of the meta-learner's trajectory distribution. We formulate unsupervised
meta-RL as information maximization between a latent task variable and the
meta-learner's data distribution, and describe a practical instantiation which
alternates between integration of recent experience into the task distribution
and meta-learning of the updated tasks. Repeating this procedure leads to
iterative reorganization such that the curriculum adapts as the meta-learner's
data distribution shifts. In particular, we show how discriminative clustering
for visual representation can support trajectory-level task acquisition and
exploration in domains with pixel observations, avoiding pitfalls of
alternatives. In experiments on vision-based navigation and manipulation
domains, we show that the algorithm allows for unsupervised meta-learning that
transfers to downstream tasks specified by hand-crafted reward functions and
serves as pre-training for more efficient supervised meta-learning of test task
distributions.
