The Unusual Effectiveness of Averaging in GAN Training
  We examine two different techniques for parameter averaging in GAN training.
Moving Average (MA) computes the time-average of parameters, whereas
Exponential Moving Average (EMA) computes an exponentially discounted sum.
Whilst MA is known to lead to convergence in bilinear settings, we provide the
-- to our knowledge -- first theoretical arguments in support of EMA. We show
that EMA converges to limit cycles around the equilibrium with vanishing
amplitude as the discount parameter approaches one for simple bilinear games
and also enhances the stability of general GAN training. We establish
experimentally that both techniques are strikingly effective in the
non-convex-concave GAN setting as well. Both improve inception and FID scores
on different architectures and for different GAN objectives. We provide
comprehensive experimental results across a range of datasets -- mixture of
Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its
effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce
clean CelebA face images.\footnote{~The code is available at
\url{https://github.com/yasinyazici/EMA_GAN}}
