EvalNorm: Estimating Batch Normalization Statistics for Evaluation
  Batch normalization (BN) has been very effective for deep learning and is
widely used. However, when training with small minibatches, models using BN
exhibit a significant degradation in performance. In this paper we study this
peculiar behavior of BN to gain a better understanding of the problem, and
identify a cause. We propose 'EvalNorm' to address the issue by estimating
corrected normalization statistics to use for BN during evaluation. EvalNorm
supports online estimation of the corrected statistics while the model is being
trained, and does not affect the training scheme of the model. As a result,
EvalNorm can also be used with existing pre-trained models allowing them to
benefit from our method. EvalNorm yields large gains for models trained with
smaller batches. Our experiments show that EvalNorm performs 6.18% (absolute)
better than vanilla BN for a batchsize of 2 on ImageNet validation set and from
1.5 to 7.0 points (absolute) gain on the COCO object detection benchmark across
a variety of setups.
