Learning to Walk via Deep Reinforcement Learning
  Deep reinforcement learning (deep RL) holds the promise of automating the
acquisition of complex controllers that can map sensory inputs directly to
low-level actions. In the domain of robotic locomotion, deep RL could enable
learning locomotion skills with minimal engineering and without an explicit
model of the robot dynamics. Unfortunately, applying deep RL to real-world
robotic tasks is exceptionally difficult, primarily due to poor sample
complexity and sensitivity to hyperparameters. While hyperparameters can be
easily tuned in simulated domains, tuning may be prohibitively expensive on
physical systems, such as legged robots, that can be damaged through extensive
trial-and-error learning. In this paper, we propose a sample-efficient deep RL
algorithm based on maximum entropy RL that requires minimal per-task tuning and
only a modest number of trials to learn neural network policies. We apply this
method to learning walking gaits on a real-world Minitaur robot. Our method can
acquire a stable gait from scratch directly in the real world in about two
hours, without relying on any model or simulation, and the resulting policy is
robust to moderate variations in the environment. We further show that our
algorithm achieves state-of-the-art performance on simulated benchmarks with a
single set of hyperparameters. Videos of training and the learned policy can be
found on the project website.
