Scaling Limit of Neural Networks with the Xavier Initialization and
  Convergence to a Global Minimum
  We analyze single-layer neural networks with the Xavier initialization in the
asymptotic regime of large numbers of hidden units and large numbers of
stochastic gradient descent training steps. The evolution of the neural network
during training can be viewed as a stochastic system and, using techniques from
stochastic analysis, we prove the neural network converges in distribution to a
random ODE with a Gaussian distribution. The limit is completely different than
in the typical mean-field results for neural networks due to the
$\frac{1}{\sqrt{N}}$ normalization factor in the Xavier initialization (versus
the $\frac{1}{N}$ factor in the typical mean-field framework). Although the
pre-limit problem of optimizing a neural network is non-convex (and therefore
the neural network may converge to a local minimum), the limit equation
minimizes a (quadratic) convex objective function and therefore converges to a
global minimum. Furthermore, under reasonable assumptions, the matrix in the
limiting quadratic objective function is positive definite and thus the neural
network (in the limit) will converge to a global minimum with zero loss on the
training set.
