On Proximal Policy Optimization's Heavy-tailed Gradients
  Modern policy gradient algorithms such as Proximal Policy Optimization (PPO)
rely on an arsenal of heuristics, including loss clipping and gradient
clipping, to ensure successful learning. These heuristics are reminiscent of
techniques from robust statistics, commonly used for estimation in outlier-rich
(``heavy-tailed'') regimes. In this paper, we present a detailed empirical
study to characterize the heavy-tailed nature of the gradients of the PPO
surrogate reward function. We demonstrate that the gradients, especially for
the actor network, exhibit pronounced heavy-tailedness and that it increases as
the agent's policy diverges from the behavioral policy (i.e., as the agent goes
further off policy). Further examination implicates the likelihood ratios and
advantages in the surrogate reward as the main sources of the observed
heavy-tailedness. We then highlight issues arising due to the heavy-tailed
nature of the gradients. In this light, we study the effects of the standard
PPO clipping heuristics, demonstrating that these tricks primarily serve to
offset heavy-tailedness in gradients. Thus motivated, we propose incorporating
GMOM, a high-dimensional robust estimator, into PPO as a substitute for three
clipping tricks. Despite requiring less hyperparameter tuning, our method
matches the performance of PPO (with all heuristics enabled) on a battery of
MuJoCo continuous control tasks.
