Micro-Batch Training with Batch-Channel Normalization and Weight
  Standardization
  Batch Normalization (BN) has become an out-of-box technique to improve deep
network training. However, its effectiveness is limited for micro-batch
training, i.e., each GPU typically has only 1-2 images for training, which is
inevitable for many computer vision tasks, e.g., object detection and semantic
segmentation, constrained by memory consumption. To address this issue, we
propose Weight Standardization (WS) and Batch-Channel Normalization (BCN) to
bring two success factors of BN into micro-batch training: 1) the smoothing
effects on the loss landscape and 2) the ability to avoid harmful elimination
singularities along the training trajectory. WS standardizes the weights in
convolutional layers to smooth the loss landscape by reducing the Lipschitz
constants of the loss and the gradients; BCN combines batch and channel
normalizations and leverages estimated statistics of the activations in
convolutional layers to keep networks away from elimination singularities. We
validate WS and BCN on comprehensive computer vision tasks, including image
classification, object detection, instance segmentation, video recognition and
semantic segmentation. All experimental results consistently show that WS and
BCN improve micro-batch training significantly. Moreover, using WS and BCN with
micro-batch training is even able to match or outperform the performances of BN
with large-batch training.
