Overcoming Oscillations in Quantization-Aware Training
  When training neural networks with simulated quantization, we observe that
quantized weights can, rather unexpectedly, oscillate between two grid-points.
The importance of this effect and its impact on quantization-aware training are
not well-understood or investigated in literature. In this paper, we delve
deeper into the phenomenon of weight oscillations and show that it can lead to
a significant accuracy degradation due to wrongly estimated batch-normalization
statistics during inference and increased noise during training. These effects
are particularly pronounced in low-bit ($\leq$ 4-bits) quantization of
efficient networks with depth-wise separable layers, such as MobileNets and
EfficientNets. In our analysis we investigate several previously proposed
quantization-aware training (QAT) algorithms and show that most of these are
unable to overcome oscillations. Finally, we propose two novel QAT algorithms
to overcome oscillations during training: oscillation dampening and iterative
weight freezing. We demonstrate that our algorithms achieve state-of-the-art
accuracy for low-bit (3 & 4 bits) weight and activation quantization of
efficient architectures, such as MobileNetV2, MobileNetV3, and EfficentNet-lite
on ImageNet.
