Towards Understanding Regularization in Batch Normalization
  Batch Normalization (BN) improves both convergence and generalization in
training neural networks. This work understands these phenomena theoretically.
We analyze BN by using a basic block of neural networks, consisting of a kernel
layer, a BN layer, and a nonlinear activation function. This basic network
helps us understand the impacts of BN in three aspects. First, by viewing BN as
an implicit regularizer, BN can be decomposed into population normalization
(PN) and gamma decay as an explicit regularization. Second, learning dynamics
of BN and the regularization show that training converged with large maximum
and effective learning rate. Third, generalization of BN is explored by using
statistical mechanics. Experiments demonstrate that BN in convolutional neural
networks share the same traits of regularization as the above analyses.
