Adaptive Precision Training (AdaPT): A dynamic fixed point quantized
  training approach for DNNs
  Quantization is a technique for reducing deep neural networks (DNNs) training
and inference times, which is crucial for training in resource constrained
environments or applications where inference is time critical. State-of-the-art
(SOTA) quantization approaches focus on post-training quantization, i.e.,
quantization of pre-trained DNNs for speeding up inference. While work on
quantized training exists, most approaches require refinement in full precision
(usually single precision) in the final training phase or enforce a global word
length across the entire DNN. This leads to suboptimal assignments of
bit-widths to layers and, consequently, suboptimal resource usage. In an
attempt to overcome such limitations, we introduce AdaPT, a new fixed-point
quantized sparsifying training strategy. AdaPT decides about precision switches
between training epochs based on information theoretic conditions. The goal is
to determine on a per-layer basis the lowest precision that causes no
quantization-induced information loss while keeping the precision high enough
such that future learning steps do not suffer from vanishing gradients. The
benefits of the resulting fully quantized DNN are evaluated based on an
analytical performance model which we develop. We illustrate that an average
speedup of 1.27 compared to standard training in float32 with an average
accuracy increase of 0.98% can be achieved for AlexNet/ResNet on CIFAR10/100
and we further demonstrate these AdaPT trained models achieve an average
inference speedup of 2.33 with a model size reduction of 0.52.
