Global Convergence of Second-order Dynamics in Two-layer Neural Networks
  Recent results have shown that for two-layer fully connected neural networks,
gradient flow converges to a global optimum in the infinite width limit, by
making a connection between the mean field dynamics and the Wasserstein
gradient flow. These results were derived for first-order gradient flow, and a
natural question is whether second-order dynamics, i.e., dynamics with
momentum, exhibit a similar guarantee. We show that the answer is positive for
the heavy ball method. In this case, the resulting integro-PDE is a nonlinear
kinetic Fokker Planck equation, and unlike the first-order case, it has no
apparent connection with the Wasserstein gradient flow. Instead, we study the
variations of a Lyapunov functional along the solution trajectories to
characterize the stationary points and to prove convergence. While our results
are asymptotic in the mean field limit, numerical simulations indicate that
global convergence may already occur for reasonably small networks.
