Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification,
  and Local Computations
  Communication bottleneck has been identified as a significant issue in
distributed optimization of large-scale learning models. Recently, several
approaches to mitigate this problem have been proposed, including different
forms of gradient compression or computing local models and mixing them
iteratively. In this paper, we propose \emph{Qsparse-local-SGD} algorithm,
which combines aggressive sparsification with quantization and local
computation along with error compensation, by keeping track of the difference
between the true and compressed gradients. We propose both synchronous and
asynchronous implementations of \emph{Qsparse-local-SGD}. We analyze
convergence for \emph{Qsparse-local-SGD} in the \emph{distributed} setting for
smooth non-convex and convex objective functions. We demonstrate that
\emph{Qsparse-local-SGD} converges at the same rate as vanilla distributed SGD
for many important classes of sparsifiers and quantizers. We use
\emph{Qsparse-local-SGD} to train ResNet-50 on ImageNet and show that it
results in significant savings over the state-of-the-art, in the number of bits
transmitted to reach target accuracy.
