Towards Unified INT8 Training for Convolutional Neural Network
  Recently low-bit (e.g., 8-bit) network quantization has been extensively
studied to accelerate the inference. Besides inference, low-bit training with
quantized gradients can further bring more considerable acceleration, since the
backward process is often computation-intensive. Unfortunately, the
inappropriate quantization of backward propagation usually makes the training
unstable and even crash. There lacks a successful unified low-bit training
framework that can support diverse networks on various tasks. In this paper, we
give an attempt to build a unified 8-bit (INT8) training framework for common
convolutional neural networks from the aspects of both accuracy and speed.
First, we empirically find the four distinctive characteristics of gradients,
which provide us insightful clues for gradient quantization. Then, we
theoretically give an in-depth analysis of the convergence bound and derive two
principles for stable INT8 training. Finally, we propose two universal
techniques, including Direction Sensitive Gradient Clipping that reduces the
direction deviation of gradients and Deviation Counteractive Learning Rate
Scaling that avoids illegal gradient update along the wrong direction. The
experiments show that our unified solution promises accurate and efficient INT8
training for a variety of networks and tasks, including MobileNetV2,
InceptionV3 and object detection that prior studies have never succeeded.
Moreover, it enjoys a strong flexibility to run on off-the-shelf hardware, and
reduces the training time by 22% on Pascal GPU without too much optimization
effort. We believe that this pioneering study will help lead the community
towards a fully unified INT8 training for convolutional neural networks.
