Spectral Normalisation for Deep Reinforcement Learning: an Optimisation
  Perspective
  Most of the recent deep reinforcement learning advances take an RL-centric
perspective and focus on refinements of the training objective. We diverge from
this view and show we can recover the performance of these developments not by
changing the objective, but by regularising the value-function estimator.
Constraining the Lipschitz constant of a single layer using spectral
normalisation is sufficient to elevate the performance of a Categorical-DQN
agent to that of a more elaborated \rainbow{} agent on the challenging Atari
domain. We conduct ablation studies to disentangle the various effects
normalisation has on the learning dynamics and show that is sufficient to
modulate the parameter updates to recover most of the performance of spectral
normalisation. These findings hint towards the need to also focus on the neural
component and its learning dynamics to tackle the peculiarities of Deep
Reinforcement Learning.
