Towards Efficient Full 8-bit Integer DNN Online Training on
  Resource-limited Devices without Batch Normalization
  Huge computational costs brought by convolution and batch normalization (BN)
have caused great challenges for the online training and corresponding
applications of deep neural networks (DNNs), especially in resource-limited
devices. Existing works only focus on the convolution or BN acceleration and no
solution can alleviate both problems with satisfactory performance. Online
training has gradually become a trend in resource-limited devices like mobile
phones while there is still no complete technical scheme with acceptable model
performance, processing speed, and computational cost. In this research, an
efficient online-training quantization framework termed EOQ is proposed by
combining Fixup initialization and a novel quantization scheme for DNN model
compression and acceleration. Based on the proposed framework, we have
successfully realized full 8-bit integer network training and removed BN in
large-scale DNNs. Especially, weight updates are quantized to 8-bit integers
for the first time. Theoretical analyses of EOQ utilizing Fixup initialization
for removing BN have been further given using a novel Block Dynamical Isometry
theory with weaker assumptions. Benefiting from rational quantization
strategies and the absence of BN, the full 8-bit networks based on EOQ can
achieve state-of-the-art accuracy and immense advantages in computational cost
and processing speed. What is more, the design of deep learning chips can be
profoundly simplified for the absence of unfriendly square root operations in
BN. Beyond this, EOQ has been evidenced to be more advantageous in small-batch
online training with fewer batch samples. In summary, the EOQ framework is
specially designed for reducing the high cost of convolution and BN in network
training, demonstrating a broad application prospect of online training in
resource-limited devices.
