Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent
  Optimization in Policy Gradient Methods
  Recent analyses of certain gradient descent optimization methods have shown
that performance can degrade in some settings - such as with stochasticity or
implicit momentum. In deep reinforcement learning (Deep RL), such optimization
methods are often used for training neural networks via the temporal difference
error or policy gradient. As an agent improves over time, the optimization
target changes and thus the loss landscape (and local optima) change. Due to
the failure modes of those methods, the ideal choice of optimizer for Deep RL
remains unclear. As such, we provide an empirical analysis of the effects that
a wide range of gradient descent optimizers and their hyperparameters have on
policy gradient methods, a subset of Deep RL algorithms, for benchmark
continuous control tasks. We find that adaptive optimizers have a narrow window
of effective learning rates, diverging in other cases, and that the
effectiveness of momentum varies depending on the properties of the
environment. Our analysis suggests that there is significant interplay between
the dynamics of the environment and Deep RL algorithm properties which aren't
necessarily accounted for by traditional adaptive gradient methods. We provide
suggestions for optimal settings of current methods and further lines of
research based on our findings.
