Adaptive Gradient Quantization for Data-Parallel SGD
  Many communication-efficient variants of SGD use gradient quantization
schemes. These schemes are often heuristic and fixed over the course of
training. We empirically observe that the statistics of gradients of deep
models change during the training. Motivated by this observation, we introduce
two adaptive quantization schemes, ALQ and AMQ. In both schemes, processors
update their compression schemes in parallel by efficiently computing
sufficient statistics of a parametric distribution. We improve the validation
accuracy by almost 2% on CIFAR-10 and 1% on ImageNet in challenging low-cost
communication setups. Our adaptive methods are also significantly more robust
to the choice of hyperparameters.
