Ctrl-Z: Recovering from Instability in Reinforcement Learning
  When learning behavior, training data is often generated by the learner
itself; this can result in unstable training dynamics, and this problem has
particularly important applications in safety-sensitive real-world control
tasks such as robotics. In this work, we propose a principled and
model-agnostic approach to mitigate the issue of unstable learning dynamics by
maintaining a history of a reinforcement learning agent over the course of
training, and reverting to the parameters of a previous agent whenever
performance significantly decreases. We develop techniques for evaluating this
performance through statistical hypothesis testing of continued improvement,
and evaluate them on a standard suite of challenging benchmark tasks involving
continuous control of simulated robots. We show improvements over
state-of-the-art reinforcement learning algorithms in performance and
robustness to hyperparameters, outperforming DDPG in 5 out of 6 evaluation
environments and showing no decrease in performance with TD3, which is known to
be relatively stable. In this way, our approach takes an important step towards
increasing data efficiency and stability in training for real-world robotic
applications.
