Learning to Locomote: Understanding How Environment Design Matters for
  Deep Reinforcement Learning
  Learning to locomote is one of the most common tasks in physics-based
animation and deep reinforcement learning (RL). A learned policy is the product
of the problem to be solved, as embodied by the RL environment, and the RL
algorithm. While enormous attention has been devoted to RL algorithms, much
less is known about the impact of design choices for the RL environment. In
this paper, we show that environment design matters in significant ways and
document how it can contribute to the brittle nature of many RL results.
Specifically, we examine choices related to state representations, initial
state distributions, reward structure, control frequency, episode termination
procedures, curriculum usage, the action space, and the torque limits. We aim
to stimulate discussion around such choices, which in practice strongly impact
the success of RL when applied to continuous-action control problems of
interest to animation, such as learning to locomote.
