Decentralized Local Stochastic Extra-Gradient for Variational
  Inequalities
  We consider distributed stochastic variational inequalities (VIs) on
unbounded domain with the problem data being heterogeneous (non-IID) and
distributed across many devices. We make very general assumption on the
computational network that, in particular, covers the settings of fully
decentralized calculations with time-varying networks and centralized
topologies commonly used in Federated Learning. Moreover, multiple local
updates on the workers can be made for reducing the communication frequency
between workers. We extend stochastic extragradient method to this very general
setting and theoretically analyze its convergence rate in the strongly
monotone, monotone, and non-monotone setting when an Minty solution exists. The
provided rates have explicit dependence on\ network characteristics and how it
varies with time, data heterogeneity, variance, number of devices, and other
standard parameters. As a special case, our method and analysis apply to
distributed stochastic saddle-point problems (SPP), e.g., to training Deep
Generative Adversarial Networks (GANs) for which the decentralized training has
been reported to be extremely challenging. In experiments for decentralized
training of GANs we demonstrate the effectiveness of our proposed approach.
