Continuous vs. Discrete Optimization of Deep Neural Networks
  Existing analyses of optimization in deep learning are either continuous,
focusing on (variants of) gradient flow, or discrete, directly treating
(variants of) gradient descent. Gradient flow is amenable to theoretical
analysis, but is stylized and disregards computational efficiency. The extent
to which it represents gradient descent is an open question in the theory of
deep learning. The current paper studies this question. Viewing gradient
descent as an approximate numerical solution to the initial value problem of
gradient flow, we find that the degree of approximation depends on the
curvature around the gradient flow trajectory. We then show that over deep
neural networks with homogeneous activations, gradient flow trajectories enjoy
favorable curvature, suggesting they are well approximated by gradient descent.
This finding allows us to translate an analysis of gradient flow over deep
linear neural networks into a guarantee that gradient descent efficiently
converges to global minimum almost surely under random initialization.
Experiments suggest that over simple deep neural networks, gradient descent
with conventional step size is indeed close to gradient flow. We hypothesize
that the theory of gradient flows will unravel mysteries behind deep learning.
