On the One-sided Convergence of Adam-type Algorithms in Non-convex
  Non-concave Min-max Optimization
  Adam-type methods, the extension of adaptive gradient methods, have shown
great performance in the training of both supervised and unsupervised machine
learning models. In particular, Adam-type optimizers have been widely used
empirically as the default tool for training generative adversarial networks
(GANs). On the theory side, however, despite the existence of theoretical
results showing the efficiency of Adam-type methods in minimization problems,
the reason of their wonderful performance still remains absent in GAN's
training. In existing works, the fast convergence has long been considered as
one of the most important reasons and multiple works have been proposed to give
a theoretical guarantee of the convergence to a critical point of min-max
optimization algorithms under certain assumptions. In this paper, we firstly
argue empirically that in GAN's training, Adam does not converge to a critical
point even upon successful training: Only the generator is converging while the
discriminator's gradient norm remains high throughout the training. We name
this one-sided convergence. Then we bridge the gap between experiments and
theory by showing that Adam-type algorithms provably converge to a one-sided
first order stationary points in min-max optimization problems under the
one-sided MVI condition. We also empirically verify that such one-sided MVI
condition is satisfied for standard GANs after trained over standard data sets.
To the best of our knowledge, this is the very first result which provides an
empirical observation and a strict theoretical guarantee on the one-sided
convergence of Adam-type algorithms in min-max optimization.
