GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient
  Aggregation in Distributed CNN Training
  Data parallelism can boost the training speed of convolutional neural
networks (CNN), but could suffer from significant communication costs caused by
gradient aggregation. To alleviate this problem, several scalar quantization
techniques have been developed to compress the gradients. But these techniques
could perform poorly when used together with decentralized aggregation
protocols like ring all-reduce (RAR), mainly due to their inability to directly
aggregate compressed gradients. In this paper, we empirically demonstrate the
strong linear correlations between CNN gradients, and propose a gradient vector
quantization technique, named GradiVeQ, to exploit these correlations through
principal component analysis (PCA) for substantial gradient dimension
reduction. GradiVeQ enables direct aggregation of compressed gradients, hence
allows us to build a distributed learning system that parallelizes GradiVeQ
gradient compression and RAR communications. Extensive experiments on popular
CNNs demonstrate that applying GradiVeQ slashes the wall-clock gradient
aggregation time of the original RAR by more than 5X without noticeable
accuracy loss, and reduces the end-to-end training time by almost 50%. The
results also show that GradiVeQ is compatible with scalar quantization
techniques such as QSGD (Quantized SGD), and achieves a much higher speed-up
gain under the same compression ratio.
