Unsupervised Meta-Learning for Reinforcement Learning
  Meta-learning algorithms use past experience to learn to quickly solve new
tasks. In the context of reinforcement learning, meta-learning algorithms
acquire reinforcement learning procedures to solve new problems more
efficiently by utilizing experience from prior tasks. The performance of
meta-learning algorithms depends on the tasks available for meta-training: in
the same way that supervised learning generalizes best to test points drawn
from the same distribution as the training points, meta-learning methods
generalize best to tasks from the same distribution as the meta-training tasks.
In effect, meta-reinforcement learning offloads the design burden from
algorithm design to task design. If we can automate the process of task design
as well, we can devise a meta-learning algorithm that is truly automated. In
this work, we take a step in this direction, proposing a family of unsupervised
meta-learning algorithms for reinforcement learning. We motivate and describe a
general recipe for unsupervised meta-reinforcement learning, and present an
instantiation of this approach. Our conceptual and theoretical contributions
consist of formulating the unsupervised meta-reinforcement learning problem and
describing how task proposals based on mutual information can be used to train
optimal meta-learners. Our experimental results indicate that unsupervised
meta-reinforcement learning effectively acquires accelerated reinforcement
learning procedures without the need for manual task design and these
procedures exceed the performance of learning from scratch.
