Which Training Methods for GANs do actually Converge?
  Recent work has shown local convergence of GAN training for absolutely
continuous data and generator distributions. In this paper, we show that the
requirement of absolute continuity is necessary: we describe a simple yet
prototypical counterexample showing that in the more realistic case of
distributions that are not absolutely continuous, unregularized GAN training is
not always convergent. Furthermore, we discuss regularization strategies that
were recently proposed to stabilize GAN training. Our analysis shows that GAN
training with instance noise or zero-centered gradient penalties converges. On
the other hand, we show that Wasserstein-GANs and WGAN-GP with a finite number
of discriminator updates per generator update do not always converge to the
equilibrium point. We discuss these results, leading us to a new explanation
for the stability problems of GAN training. Based on our analysis, we extend
our convergence results to more general GANs and prove local convergence for
simplified gradient penalties even if the generator and data distribution lie
on lower dimensional manifolds. We find these penalties to work well in
practice and use them to learn high-resolution generative image models for a
variety of datasets with little hyperparameter tuning.
