Distributed Sparse SGD with Majority Voting
  Distributed learning, particularly variants of distributed stochastic
gradient descent (DSGD), are widely employed to speed up training by leveraging
computational resources of several workers. However, in practise, communication
delay becomes a bottleneck due to the significant amount of information that
needs to be exchanged between the workers and the parameter server. One of the
most efficient strategies to mitigate the communication bottleneck is top-K
sparsification. However, top-K sparsification requires additional communication
load to represent the sparsity pattern, and the mismatch between the sparsity
patterns of the workers prevents exploitation of efficient communication
protocols. To address these issues, we introduce a novel majority voting based
sparse communication strategy, in which the workers first seek a consensus on
the structure of the sparse representation. This strategy provides a
significant reduction in the communication load and allows using the same
sparsity level in both communication directions. Through extensive simulations
on the CIFAR-10 dataset, we show that it is possible to achieve up to x4000
compression without any loss in the test accuracy.
