Gradient Sparification for Asynchronous Distributed Training
  Modern large scale machine learning applications require stochastic
optimization algorithms to be implemented on distributed computational
architectures. A key bottleneck is the communication overhead for exchanging
information, such as stochastic gradients, among different nodes. Recently,
gradient sparsification techniques have been proposed to reduce communications
cost and thus alleviate the network overhead. However, most of gradient
sparsification techniques consider only synchronous parallelism and cannot be
applied in asynchronous scenarios, such as asynchronous distributed training
for federated learning at mobile devices.
  In this paper, we present a dual-way gradient sparsification approach (DGS)
that is suitable for asynchronous distributed training. We let workers download
model difference, instead of the global model, from the server, and the model
difference information is also sparsified so that the information exchanged
overhead is reduced by sparsifying the dual-way communication between the
server and workers. To preserve accuracy under dual-way sparsification, we
design a sparsification aware momentum (SAMomentum) to turn sparsification into
adaptive batch size between each parameter. We conduct experiments at a cluster
of 32 workers, and the results show that, with the same compression ratio but
much lower communication cost, our approach can achieve better scalability and
generalization ability.
