The problem with DDPG: understanding failures in deterministic
  environments with sparse rewards
  In environments with continuous state and action spaces, state-of-the-art
actor-critic reinforcement learning algorithms can solve very complex problems,
yet can also fail in environments that seem trivial, but the reason for such
failures is still poorly understood. In this paper, we contribute a formal
explanation of these failures in the particular case of sparse reward and
deterministic environments. First, using a very elementary control problem, we
illustrate that the learning process can get stuck into a fixed point
corresponding to a poor solution. Then, generalizing from the studied example,
we provide a detailed analysis of the underlying mechanisms which results in a
new understanding of one of the convergence regimes of these algorithms. The
resulting perspective casts a new light on already existing solutions to the
issues we have highlighted, and suggests other potential approaches.
