ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring
  for Minimax Problems
  Despite remarkable empirical success, the training dynamics of generative
adversarial networks (GAN), which involves solving a minimax game using
stochastic gradients, is still poorly understood. In this work, we analyze
last-iterate convergence of simultaneous gradient descent (simGD) and its
variants under the assumption of convex-concavity, guided by a continuous-time
analysis with differential equations. First, we show that simGD, as is,
converges with stochastic sub-gradients under strict convexity in the primal
variable. Second, we generalize optimistic simGD to accommodate an optimism
rate separate from the learning rate and show its convergence with full
gradients. Finally, we present anchored simGD, a new method, and show
convergence with stochastic subgradients.
