Communication-Compressed Adaptive Gradient Method for Distributed
  Nonconvex Optimization
  Due to the explosion in the size of the training datasets, distributed
learning has received growing interest in recent years. One of the major
bottlenecks is the large communication cost between the central server and the
local workers. While error feedback compression has been proven to be
successful in reducing communication costs with stochastic gradient descent
(SGD), there are much fewer attempts in building communication-efficient
adaptive gradient methods with provable guarantees, which are widely used in
training large-scale machine learning models. In this paper, we propose a new
communication-compressed AMSGrad for distributed nonconvex optimization
problem, which is provably efficient. Our proposed distributed learning
framework features an effective gradient compression strategy and a worker-side
model update design. We prove that the proposed communication-efficient
distributed adaptive gradient method converges to the first-order stationary
point with the same iteration complexity as uncompressed vanilla AMSGrad in the
stochastic nonconvex optimization setting. Experiments on various benchmarks
back up our theory.
