Improving Neural Network Quantization without Retraining using Outlier
  Channel Splitting
  Quantization can improve the execution latency and energy efficiency of
neural networks on both commodity GPUs and specialized accelerators. The
majority of existing literature focuses on training quantized DNNs, while this
work examines the less-studied topic of quantizing a floating-point model
without (re)training. DNN weights and activations follow a bell-shaped
distribution post-training, while practical hardware uses a linear quantization
grid. This leads to challenges in dealing with outliers in the distribution.
Prior work has addressed this by clipping the outliers or using specialized
hardware. In this work, we propose outlier channel splitting (OCS), which
duplicates channels containing outliers, then halves the channel values. The
network remains functionally identical, but affected outliers are moved toward
the center of the distribution. OCS requires no additional training and works
on commodity hardware. Experimental evaluation on ImageNet classification and
language modeling shows that OCS can outperform state-of-the-art clipping
techniques with only minor overhead.
