Local Stochastic Gradient Descent Ascent: Convergence Analysis and
  Communication Efficiency
  Local SGD is a promising approach to overcome the communication overhead in
distributed learning by reducing the synchronization frequency among worker
nodes. Despite the recent theoretical advances of local SGD in empirical risk
minimization, the efficiency of its counterpart in minimax optimization remains
unexplored. Motivated by large scale minimax learning problems, such as
adversarial robust learning and training generative adversarial networks
(GANs), we propose local Stochastic Gradient Descent Ascent (local SGDA), where
the primal and dual variables can be trained locally and averaged periodically
to significantly reduce the number of communications. We show that local SGDA
can provably optimize distributed minimax problems in both homogeneous and
heterogeneous data with reduced number of communications and establish
convergence rates under strongly-convex-strongly-concave and
nonconvex-strongly-concave settings. In addition, we propose a novel variant
local SGDA+, to solve nonconvex-nonconcave problems. We give corroborating
empirical evidence on different distributed minimax problems.
