L1-Norm Batch Normalization for Efficient Training of Deep Neural
  Networks
  Batch Normalization (BN) has been proven to be quite effective at
accelerating and improving the training of deep neural networks (DNNs).
However, BN brings additional computation, consumes more memory and generally
slows down the training process by a large margin, which aggravates the
training effort. Furthermore, the nonlinear square and root operations in BN
also impede the low bit-width quantization techniques, which draws much
attention in deep learning hardware community. In this work, we propose an
L1-norm BN (L1BN) with only linear operations in both the forward and the
backward propagations during training. L1BN is shown to be approximately
equivalent to the original L2-norm BN (L2BN) by multiplying a scaling factor.
Experiments on various convolutional neural networks (CNNs) and generative
adversarial networks (GANs) reveal that L1BN maintains almost the same
accuracies and convergence rates compared to L2BN but with higher computational
efficiency. On FPGA platform, the proposed signum and absolute operations in
L1BN can achieve 1.5$\times$ speedup and save 50\% power consumption, compared
with the original costly square and root operations, respectively. This
hardware-friendly normalization method not only surpasses L2BN in speed, but
also simplify the hardware design of ASIC accelerators with higher energy
efficiency. Last but not the least, L1BN promises a fully quantized training of
DNNs, which is crucial to future adaptive terminal devices.
