Mixed Precision Training With 8-bit Floating Point
  Reduced precision computation for deep neural networks is one of the key
areas addressing the widening compute gap driven by an exponential growth in
model size. In recent years, deep learning training has largely migrated to
16-bit precision, with significant gains in performance and energy efficiency.
However, attempts to train DNNs at 8-bit precision have met with significant
challenges because of the higher precision and dynamic range requirements of
back-propagation. In this paper, we propose a method to train deep neural
networks using 8-bit floating point representation for weights, activations,
errors, and gradients. In addition to reducing compute precision, we also
reduced the precision requirements for the master copy of weights from 32-bit
to 16-bit. We demonstrate state-of-the-art accuracy across multiple data sets
(imagenet-1K, WMT16) and a broader set of workloads (Resnet-18/34/50, GNMT,
Transformer) than previously reported. We propose an enhanced loss scaling
method to augment the reduced subnormal range of 8-bit floating point for
improved error propagation. We also examine the impact of quantization noise on
generalization and propose a stochastic rounding technique to address gradient
noise. As a result of applying all these techniques, we report slightly higher
validation accuracy compared to full precision baseline.
