Sparsified SGD with Memory
  Huge scale machine learning problems are nowadays tackled by distributed
optimization algorithms, i.e. algorithms that leverage the compute power of
many devices for training. The communication overhead is a key bottleneck that
hinders perfect scalability. Various recent works proposed to use quantization
or sparsification techniques to reduce the amount of data that needs to be
communicated, for instance by only sending the most significant entries of the
stochastic gradient (top-k sparsification). Whilst such schemes showed very
promising performance in practice, they have eluded theoretical analysis so
far.
  In this work we analyze Stochastic Gradient Descent (SGD) with
k-sparsification or compression (for instance top-k or random-k) and show that
this scheme converges at the same rate as vanilla SGD when equipped with error
compensation (keeping track of accumulated errors in memory). That is,
communication can be reduced by a factor of the dimension of the problem
(sometimes even more) whilst still converging at the same rate. We present
numerical experiments to illustrate the theoretical findings and the better
scalability for distributed applications.
