Meta-Reinforcement Learning of Structured Exploration Strategies
  Exploration is a fundamental challenge in reinforcement learning (RL). Many
of the current exploration methods for deep RL use task-agnostic objectives,
such as information gain or bonuses based on state visitation. However, many
practical applications of RL involve learning more than a single task, and
prior tasks can be used to inform how exploration should be performed in new
tasks. In this work, we explore how prior tasks can inform an agent about how
to explore effectively in new situations. We introduce a novel gradient-based
fast adaptation algorithm -- model agnostic exploration with structured noise
(MAESN) -- to learn exploration strategies from prior experience. The prior
experience is used both to initialize a policy and to acquire a latent
exploration space that can inject structured stochasticity into a policy,
producing exploration strategies that are informed by prior knowledge and are
more effective than random action-space noise. We show that MAESN is more
effective at learning exploration strategies when compared to prior meta-RL
methods, RL without learned exploration strategies, and task-agnostic
exploration methods. We evaluate our method on a variety of simulated tasks:
locomotion with a wheeled robot, locomotion with a quadrupedal walker, and
object manipulation.
