rTop-k: A Statistical Estimation Approach to Distributed SGD
  The large communication cost for exchanging gradients between different nodes
significantly limits the scalability of distributed training for large-scale
learning models. Motivated by this observation, there has been significant
recent interest in techniques that reduce the communication cost of distributed
Stochastic Gradient Descent (SGD), with gradient sparsification techniques such
as top-k and random-k shown to be particularly effective. The same observation
has also motivated a separate line of work in distributed statistical
estimation theory focusing on the impact of communication constraints on the
estimation efficiency of different statistical models. The primary goal of this
paper is to connect these two research lines and demonstrate how statistical
estimation models and their analysis can lead to new insights in the design of
communication-efficient training techniques. We propose a simple statistical
estimation model for the stochastic gradients which captures the sparsity and
skewness of their distribution. The statistically optimal communication scheme
arising from the analysis of this model leads to a new sparsification technique
for SGD, which concatenates random-k and top-k, considered separately in the
prior literature. We show through extensive experiments on both image and
language domains with CIFAR-10, ImageNet, and Penn Treebank datasets that the
concatenated application of these two sparsification methods consistently and
significantly outperforms either method applied alone.
