Variance-based Gradient Compression for Efficient Distributed Deep
  Learning
  Due to the substantial computational cost, training state-of-the-art deep
neural networks for large-scale datasets often requires distributed training
using multiple computation workers. However, by nature, workers need to
frequently communicate gradients, causing severe bottlenecks, especially on
lower bandwidth connections. A few methods have been proposed to compress
gradient for efficient communication, but they either suffer a low compression
ratio or significantly harm the resulting model accuracy, particularly when
applied to convolutional neural networks. To address these issues, we propose a
method to reduce the communication overhead of distributed deep learning. Our
key observation is that gradient updates can be delayed until an unambiguous
(high amplitude, low variance) gradient has been calculated. We also present an
efficient algorithm to compute the variance with negligible additional cost. We
experimentally show that our method can achieve very high compression ratio
while maintaining the result model accuracy. We also analyze the efficiency
using computation and communication cost models and provide the evidence that
this method enables distributed deep learning for many scenarios with commodity
environments.
