MQBench: Towards Reproducible and Deployable Model Quantization
  Benchmark
  Model quantization has emerged as an indispensable technique to accelerate
deep learning inference. While researchers continue to push the frontier of
quantization algorithms, existing quantization work is often unreproducible and
undeployable. This is because researchers do not choose consistent training
pipelines and ignore the requirements for hardware deployments. In this work,
we propose Model Quantization Benchmark (MQBench), a first attempt to evaluate,
analyze, and benchmark the reproducibility and deployability for model
quantization algorithms. We choose multiple different platforms for real-world
deployments, including CPU, GPU, ASIC, DSP, and evaluate extensive
state-of-the-art quantization algorithms under a unified training pipeline.
MQBench acts like a bridge to connect the algorithm and the hardware. We
conduct a comprehensive analysis and find considerable intuitive or
counter-intuitive insights. By aligning the training settings, we find existing
algorithms have about the same performance on the conventional academic track.
While for the hardware-deployable quantization, there is a huge accuracy gap
which remains unsettled. Surprisingly, no existing algorithm wins every
challenge in MQBench, and we hope this work could inspire future research
directions.
