Reset-Free Guided Policy Search: Efficient Deep Reinforcement Learning
  with Stochastic Initial States
  Autonomous learning of robotic skills can allow general-purpose robots to
learn wide behavioral repertoires without requiring extensive manual
engineering. However, robotic skill learning methods typically make one of
several trade-offs to enable practical real-world learning, such as requiring
manually designed policy or value function representations, initialization from
human-provided demonstrations, instrumentation of the training environment, or
extremely long training times. In this paper, we propose a new reinforcement
learning algorithm for learning manipulation skills that can train
general-purpose neural network policies with minimal human engineering, while
still allowing for fast, efficient learning in stochastic environments. Our
approach builds on the guided policy search (GPS) algorithm, which transforms
the reinforcement learning problem into supervised learning from a
computational teacher (without human demonstrations). In contrast to prior GPS
methods, which require a consistent set of initial states to which the system
must be reset after each episode, our approach can handle randomized initial
states, allowing it to be used in environments where deterministic resets are
impossible. We compare our method to existing policy search techniques in
simulation, showing that it can train high-dimensional neural network policies
with the same sample efficiency as prior GPS methods, and present real-world
results on a PR2 robotic manipulator.
