Quantity vs. Quality: On Hyperparameter Optimization for Deep
  Reinforcement Learning
  Reinforcement learning algorithms can show strong variation in performance
between training runs with different random seeds. In this paper we explore how
this affects hyperparameter optimization when the goal is to find
hyperparameter settings that perform well across random seeds. In particular,
we benchmark whether it is better to explore a large quantity of hyperparameter
settings via pruning of bad performers, or if it is better to aim for quality
of collected results by using repetitions. For this we consider the Successive
Halving, Random Search, and Bayesian Optimization algorithms, the latter two
with and without repetitions. We apply these to tuning the PPO2 algorithm on
the Cartpole balancing task and the Inverted Pendulum Swing-up task. We
demonstrate that pruning may negatively affect the optimization and that
repeated sampling does not help in finding hyperparameter settings that perform
better across random seeds. From our experiments we conclude that Bayesian
optimization with a noise robust acquisition function is the best choice for
hyperparameter optimization in reinforcement learning tasks.
