Taming GANs with Lookahead-Minmax
  Generative Adversarial Networks are notoriously challenging to train. The
underlying minmax optimization is highly susceptible to the variance of the
stochastic gradient and the rotational component of the associated game vector
field. To tackle these challenges, we propose the Lookahead algorithm for
minmax optimization, originally developed for single objective minimization
only. The backtracking step of our Lookahead-minmax naturally handles the
rotational game dynamics, a property which was identified to be key for
enabling gradient ascent descent methods to converge on challenging examples
often analyzed in the literature. Moreover, it implicitly handles high variance
without using large mini-batches, known to be essential for reaching state of
the art performance. Experimental results on MNIST, SVHN, CIFAR-10, and
ImageNet demonstrate a clear advantage of combining Lookahead-minmax with Adam
or extragradient, in terms of performance and improved stability, for
negligible memory and computational cost. Using 30-fold fewer parameters and
16-fold smaller minibatches we outperform the reported performance of the
class-dependent BigGAN on CIFAR-10 by obtaining FID of 12.19 without using the
class labels, bringing state-of-the-art GAN training within reach of common
computational resources.
