Quantized Epoch-SGD for Communication-Efficient Distributed Learning
  Due to its efficiency and ease to implement, stochastic gradient descent
(SGD) has been widely used in machine learning. In particular, SGD is one of
the most popular optimization methods for distributed learning. Recently,
quantized SGD (QSGD), which adopts quantization to reduce the communication
cost in SGD-based distributed learning, has attracted much attention. Although
several QSGD methods have been proposed, some of them are heuristic without
theoretical guarantee, and others have high quantization variance which makes
the convergence become slow. In this paper, we propose a new method, called
Quantized Epoch-SGD (QESGD), for communication-efficient distributed learning.
QESGD compresses (quantizes) the parameter with variance reduction, so that it
can get almost the same performance as that of SGD with less communication
cost. QESGD is implemented on the Parameter Server framework, and empirical
results on distributed deep learning show that QESGD can outperform other
state-of-the-art quantization methods to achieve the best performance.
