A Variational Inequality Perspective on Generative Adversarial Networks
  Generative adversarial networks (GANs) form a generative modeling approach
known for producing appealing samples, but they are notably difficult to train.
One common way to tackle this issue has been to propose new formulations of the
GAN objective. Yet, surprisingly few studies have looked at optimization
methods designed for this adversarial training. In this work, we cast GAN
optimization problems in the general variational inequality framework. Tapping
into the mathematical programming literature, we counter some common
misconceptions about the difficulties of saddle point optimization and propose
to extend techniques designed for variational inequalities to the training of
GANs. We apply averaging, extrapolation and a computationally cheaper variant
that we call extrapolation from the past to the stochastic gradient method
(SGD) and Adam.
