Implicit competitive regularization in GANs
  To improve the stability of GAN training we need to understand why they can
produce realistic samples. Presently, this is attributed to properties of the
divergence obtained under an optimal discriminator. This argument has a
fundamental flaw: If we do not impose regularity of the discriminator, it can
exploit visually imperceptible errors of the generator to always achieve the
maximal generator loss. In practice, gradient penalties are used to regularize
the discriminator. However, this needs a metric on the space of images that
captures visual similarity. Such a metric is not known, which explains the
limited success of gradient penalties in stabilizing GANs. We argue that the
performance of GANs is instead due to the implicit competitive regularization
(ICR) arising from the simultaneous optimization of generator and
discriminator. ICR promotes solutions that look real to the discriminator and
thus leverages its inductive biases to generate realistic images. We show that
opponent-aware modelling of generator and discriminator, as present in
competitive gradient descent (CGD), can significantly strengthen ICR and thus
stabilize GAN training without explicit regularization. In our experiments, we
use an existing implementation of WGAN-GP and show that by training it with CGD
we can improve the inception score (IS) on CIFAR10 for a wide range of
scenarios, without any hyperparameter tuning. The highest IS is obtained by
combining CGD with the WGAN-loss, without any explicit regularization.
