RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise
  Mixed Schemes and Multiple Precisions
  This work proposes a novel Deep Neural Network (DNN) quantization framework,
namely RMSMP, with a Row-wise Mixed-Scheme and Multi-Precision approach.
Specifically, this is the first effort to assign mixed quantization schemes and
multiple precisions within layers -- among rows of the DNN weight matrix, for
simplified operations in hardware inference, while preserving accuracy.
Furthermore, this paper makes a different observation from the prior work that
the quantization error does not necessarily exhibit the layer-wise sensitivity,
and actually can be mitigated as long as a certain portion of the weights in
every layer are in higher precisions. This observation enables layer-wise
uniformality in the hardware implementation towards guaranteed inference
acceleration, while still enjoying row-wise flexibility of mixed schemes and
multiple precisions to boost accuracy. The candidates of schemes and precisions
are derived practically and effectively with a highly hardware-informative
strategy to reduce the problem search space. With the offline determined ratio
of different quantization schemes and precisions for all the layers, the RMSMP
quantization algorithm uses the Hessian and variance-based method to
effectively assign schemes and precisions for each row. The proposed RMSMP is
tested for the image classification and natural language processing (BERT)
applications and achieves the best accuracy performance among state-of-the-arts
under the same equivalent precisions. The RMSMP is implemented on FPGA devices,
achieving 3.65x speedup in the end-to-end inference time for ResNet-18 on
ImageNet, compared with the 4-bit Fixed-point baseline.
