Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient
  Methods
  Stochastic Gradient Descent-Ascent (SGDA) is one of the most prominent
algorithms for solving min-max optimization and variational inequalities
problems (VIP) appearing in various machine learning tasks. The success of the
method led to several advanced extensions of the classical SGDA, including
variants with arbitrary sampling, variance reduction, coordinate randomization,
and distributed variants with compression, which were extensively studied in
the literature, especially during the last few years. In this paper, we propose
a unified convergence analysis that covers a large variety of stochastic
gradient descent-ascent methods, which so far have required different
intuitions, have different applications and have been developed separately in
various communities. A key to our unified framework is a parametric assumption
on the stochastic estimates. Via our general theoretical framework, we either
recover the sharpest known rates for the known special cases or tighten them.
Moreover, to illustrate the flexibility of our approach we develop several new
variants of SGDA such as a new variance-reduced method (L-SVRGDA), new
distributed methods with compression (QSGDA, DIANA-SGDA, VR-DIANA-SGDA), and a
new method with coordinate randomization (SEGA-SGDA). Although variants of the
new methods are known for solving minimization problems, they were never
considered or analyzed for solving min-max problems and VIPs. We also
demonstrate the most important properties of the new methods through extensive
numerical experiments.
