AdaGDA: Faster Adaptive Gradient Descent Ascent Methods for Minimax
  Optimization
  In the paper, we propose a class of faster adaptive Gradient Descent Ascent
(GDA) methods for solving the nonconvex-strongly-concave minimax problems by
using unified adaptive matrices, which include almost existing coordinate-wise
and global adaptive learning rates. In particular, we provide an effective
convergence analysis framework for our adaptive GDA methods. Specifically, we
propose a fast Adaptive Gradient Descent Ascent (AdaGDA) method based on the
basic momentum technique, which reaches a lower gradient complexity of
$O(\kappa^4\epsilon^{-4})$ for finding an $\epsilon$-stationary point without
large batches, which improves the existing results of the adaptive GDA methods
by a factor of $O(\sqrt{\kappa})$. At the same time, we present an accelerated
version of AdaGDA (VR-AdaGDA) method based on the momentum-based variance
reduced technique, which achieves a lower gradient complexity of
$O(\kappa^{4.5}\epsilon^{-3})$ for finding an $\epsilon$-stationary point
without large batches, which improves the existing results of the adaptive GDA
methods by a factor of $O(\epsilon^{-1})$. Moreover, we prove that our
VR-AdaGDA method can reach the best known gradient complexity of
$O(\kappa^{3}\epsilon^{-3})$ with the mini-batch size $O(\kappa^3)$. Some
experimental results on policy evaluation and fair classifier tasks verify
efficiency of our algorithms.
