Revisit Batch Normalization: New Understanding from an Optimization View
  and a Refinement via Composition Optimization
  Batch Normalization (BN) has been used extensively in deep learning to
achieve faster training process and better resulting models. However, whether
BN works strongly depends on how the batches are constructed during training
and it may not converge to a desired solution if the statistics on a batch are
not close to the statistics over the whole dataset. In this paper, we try to
understand BN from an optimization perspective by formulating the optimization
problem which motivates BN. We show when BN works and when BN does not work by
analyzing the optimization problem. We then propose a refinement of BN based on
compositional optimization techniques called Full Normalization (FN) to
alleviate the issues of BN when the batches are not constructed ideally. We
provide convergence analysis for FN and empirically study its effectiveness to
refine BN.
