Analytical Study of Momentum-Based Acceleration Methods in Paradigmatic
  High-Dimensional Non-Convex Problems
  The optimization step in many machine learning problems rarely relies on
vanilla gradient descent but it is common practice to use momentum-based
accelerated methods. Despite these algorithms being widely applied to arbitrary
loss functions, their behaviour in generically non-convex, high dimensional
landscapes is poorly understood. In this work, we use dynamical mean field
theory techniques to describe analytically the average dynamics of these
methods in a prototypical non-convex model: the (spiked) matrix-tensor model.
We derive a closed set of equations that describe the behaviour of heavy-ball
momentum and Nesterov acceleration in the infinite dimensional limit. By
numerical integration of these equations, we observe that these methods speed
up the dynamics but do not improve the algorithmic threshold with respect to
gradient descent in the spiked model.
