What Matters In On-Policy Reinforcement Learning? A Large-Scale
  Empirical Study
  In recent years, on-policy reinforcement learning (RL) has been successfully
applied to many different continuous control tasks. While RL algorithms are
often conceptually simple, their state-of-the-art implementations take numerous
low- and high-level design decisions that strongly affect the performance of
the resulting agents. Those choices are usually not extensively discussed in
the literature, leading to discrepancy between published descriptions of
algorithms and their implementations. This makes it hard to attribute progress
in RL and slows down overall progress [Engstrom'20]. As a step towards filling
that gap, we implement >50 such ``choices'' in a unified on-policy RL
framework, allowing us to investigate their impact in a large-scale empirical
study. We train over 250'000 agents in five continuous control environments of
different complexity and provide insights and practical recommendations for
on-policy training of RL agents.
