Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement
  Learning
  Deep reinforcement learning algorithms can learn complex behavioral skills,
but real-world application of these methods requires a large amount of
experience to be collected by the agent. In practical settings, such as
robotics, this involves repeatedly attempting a task, resetting the environment
between each attempt. However, not all tasks are easily or automatically
reversible. In practice, this learning process requires extensive human
intervention. In this work, we propose an autonomous method for safe and
efficient reinforcement learning that simultaneously learns a forward and reset
policy, with the reset policy resetting the environment for a subsequent
attempt. By learning a value function for the reset policy, we can
automatically determine when the forward policy is about to enter a
non-reversible state, providing for uncertainty-aware safety aborts. Our
experiments illustrate that proper use of the reset policy can greatly reduce
the number of manual resets required to learn a task, can reduce the number of
unsafe actions that lead to non-reversible states, and can automatically induce
a curriculum.
