Accelerating Neural Network Inference by Overflow Aware Quantization
  The inherent heavy computation of deep neural networks prevents their
widespread applications. A widely used method for accelerating model inference
is quantization, by replacing the input operands of a network using fixed-point
values. Then the majority of computation costs focus on the integer matrix
multiplication accumulation. In fact, high-bit accumulator leads to partially
wasted computation and low-bit one typically suffers from numerical overflow.
To address this problem, we propose an overflow aware quantization method by
designing trainable adaptive fixed-point representation, to optimize the number
of bits for each input tensor while prohibiting numeric overflow during the
computation. With the proposed method, we are able to fully utilize the
computing power to minimize the quantization loss and obtain optimized
inference performance. To verify the effectiveness of our method, we conduct
image classification, object detection, and semantic segmentation tasks on
ImageNet, Pascal VOC, and COCO datasets, respectively. Experimental results
demonstrate that the proposed method can achieve comparable performance with
state-of-the-art quantization methods while accelerating the inference process
by about 2 times.
