On the Convergence of SGD Training of Neural Networks
  Neural networks are usually trained by some form of stochastic gradient
descent (SGD)). A number of strategies are in common use intended to improve
SGD optimization, such as learning rate schedules, momentum, and batching.
These are motivated by ideas about the occurrence of local minima at different
scales, valleys, and other phenomena in the objective function. Empirical
results presented here suggest that these phenomena are not significant factors
in SGD optimization of MLP-related objective functions, and that the behavior
of stochastic gradient descent in these problems is better described as the
simultaneous convergence at different rates of many, largely non-interacting
subproblems
