FastSGD: A Fast Compressed SGD Framework for Distributed Machine
  Learning
  With the rapid increase of big data, distributed Machine Learning (ML) has
been widely applied in training large-scale models. Stochastic Gradient Descent
(SGD) is arguably the workhorse algorithm of ML. Distributed ML models trained
by SGD involve large amounts of gradient communication, which limits the
scalability of distributed ML. Thus, it is important to compress the gradients
for reducing communication. In this paper, we propose FastSGD, a Fast
compressed SGD framework for distributed ML. To achieve a high compression
ratio at a low cost, FastSGD represents the gradients as key-value pairs, and
compresses both the gradient keys and values in linear time complexity. For the
gradient value compression, FastSGD first uses a reciprocal mapper to transform
original values into reciprocal values, and then, it utilizes a logarithm
quantization to further reduce reciprocal values to small integers. Finally,
FastSGD filters reduced gradient integers by a given threshold. For the
gradient key compression, FastSGD provides an adaptive fine-grained delta
encoding method to store gradient keys with fewer bits. Extensive experiments
on practical ML models and datasets demonstrate that FastSGD achieves the
compression ratio up to 4 orders of magnitude, and accelerates the convergence
time up to 8x, compared with state-of-the-art methods.
