In-Hindsight Quantization Range Estimation for Quantized Training
  Quantization techniques applied to the inference of deep neural networks have
enabled fast and efficient execution on resource-constraint devices. The
success of quantization during inference has motivated the academic community
to explore fully quantized training, i.e. quantizing back-propagation as well.
However, effective gradient quantization is still an open problem. Gradients
are unbounded and their distribution changes significantly during training,
which leads to the need for dynamic quantization. As we show, dynamic
quantization can lead to significant memory overhead and additional data
traffic slowing down training. We propose a simple alternative to dynamic
quantization, in-hindsight range estimation, that uses the quantization ranges
estimated on previous iterations to quantize the present. Our approach enables
fast static quantization of gradients and activations while requiring only
minimal hardware support from the neural network accelerator to keep track of
output statistics in an online fashion. It is intended as a drop-in replacement
for estimating quantization ranges and can be used in conjunction with other
advances in quantized training. We compare our method to existing methods for
range estimation from the quantized training literature and demonstrate its
effectiveness with a range of architectures, including MobileNetV2, on image
classification benchmarks (Tiny ImageNet & ImageNet).
