Neural gradients are near-lognormal: improved quantized and sparse
  training
  While training can mostly be accelerated by reducing the time needed to
propagate neural gradients back throughout the model, most previous works focus
on the quantization/pruning of weights and activations. These methods are often
not applicable to neural gradients, which have very different statistical
properties. Distinguished from weights and activations, we find that the
distribution of neural gradients is approximately lognormal. Considering this,
we suggest two closed-form analytical methods to reduce the computational and
memory burdens of neural gradients. The first method optimizes the
floating-point format and scale of the gradients. The second method accurately
sets sparsity thresholds for gradient pruning. Each method achieves
state-of-the-art results on ImageNet. To the best of our knowledge, this paper
is the first to (1) quantize the gradients to 6-bit floating-point formats, or
(2) achieve up to 85% gradient sparsity -- in each case without accuracy
degradation. Reference implementation accompanies the paper.
