The Normalization Method for Alleviating Pathological Sharpness in Wide
  Neural Networks
  Normalization methods play an important role in enhancing the performance of
deep learning while their theoretical understandings have been limited. To
theoretically elucidate the effectiveness of normalization, we quantify the
geometry of the parameter space determined by the Fisher information matrix
(FIM), which also corresponds to the local shape of the loss landscape under
certain conditions. We analyze deep neural networks with random initialization,
which is known to suffer from a pathologically sharp shape of the landscape
when the network becomes sufficiently wide. We reveal that batch normalization
in the last layer contributes to drastically decreasing such pathological
sharpness if the width and sample number satisfy a specific condition. In
contrast, it is hard for batch normalization in the middle hidden layers to
alleviate pathological sharpness in many settings. We also found that layer
normalization cannot alleviate pathological sharpness either. Thus, we can
conclude that batch normalization in the last layer significantly contributes
to decreasing the sharpness induced by the FIM.
