Benchmarking Bonus-Based Exploration Methods on the Arcade Learning
  Environment
  This paper provides an empirical evaluation of recently developed exploration
algorithms within the Arcade Learning Environment (ALE). We study the use of
different reward bonuses that incentives exploration in reinforcement learning.
We do so by fixing the learning algorithm used and focusing only on the impact
of the different exploration bonuses in the agent's performance. We use
Rainbow, the state-of-the-art algorithm for value-based agents, and focus on
some of the bonuses proposed in the last few years. We consider the impact
these algorithms have on performance within the popular game Montezuma's
Revenge which has gathered a lot of interest from the exploration community,
across the the set of seven games identified by Bellemare et al. (2016) as
challenging for exploration, and easier games where exploration is not an
issue. We find that, in our setting, recently developed bonuses do not provide
significantly improved performance on Montezuma's Revenge or hard exploration
games. We also find that existing bonus-based methods may negatively impact
performance on games in which exploration is not an issue and may even perform
worse than $\epsilon$-greedy exploration.
