Preventing Value Function Collapse in Ensemble {Q}-Learning by
  Maximizing Representation Diversity
  The classic DQN algorithm is limited by the overestimation bias of the
learned Q-function. Subsequent algorithms have proposed techniques to reduce
this problem, without fully eliminating it. Recently, the Maxmin and Ensemble
Q-learning algorithms have used different estimates provided by the ensembles
of learners to reduce the overestimation bias. Unfortunately, these learners
can converge to the same point in the parametric or representation space,
falling back to the classic single neural network DQN. In this paper, we
describe a regularization technique to maximize ensemble diversity in these
algorithms. We propose and compare five regularization functions inspired from
economics theory and consensus optimization. We show that the regularized
approach significantly outperforms the Maxmin and Ensemble Q-learning
algorithms as well as non-ensemble baselines.
