Optimization Theory for ReLU Neural Networks Trained with Normalization
  Layers
  The success of deep neural networks is in part due to the use of
normalization layers. Normalization layers like Batch Normalization, Layer
Normalization and Weight Normalization are ubiquitous in practice, as they
improve generalization performance and speed up training significantly.
Nonetheless, the vast majority of current deep learning theory and non-convex
optimization literature focuses on the un-normalized setting, where the
functions under consideration do not exhibit the properties of commonly
normalized neural networks. In this paper, we bridge this gap by giving the
first global convergence result for two-layer neural networks with ReLU
activations trained with a normalization layer, namely Weight Normalization.
Our analysis shows how the introduction of normalization layers changes the
optimization landscape and can enable faster convergence as compared with
un-normalized neural networks.
