Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum
  under Heavy-Tailed Gradient Noise
  Stochastic gradient descent with momentum (SGDm) is one of the most popular
optimization algorithms in deep learning. While there is a rich theory of SGDm
for convex problems, the theory is considerably less developed in the context
of deep learning where the problem is non-convex and the gradient noise might
exhibit a heavy-tailed behavior, as empirically observed in recent studies. In
this study, we consider a \emph{continuous-time} variant of SGDm, known as the
underdamped Langevin dynamics (ULD), and investigate its asymptotic properties
under heavy-tailed perturbations. Supported by recent studies from statistical
physics, we argue both theoretically and empirically that the heavy-tails of
such perturbations can result in a bias even when the step-size is small, in
the sense that \emph{the optima of stationary distribution} of the dynamics
might not match \emph{the optima of the cost function to be optimized}. As a
remedy, we develop a novel framework, which we coin as \emph{fractional} ULD
(FULD), and prove that FULD targets the so-called Gibbs distribution, whose
optima exactly match the optima of the original cost. We observe that the Euler
discretization of FULD has noteworthy algorithmic similarities with
\emph{natural gradient} methods and \emph{gradient clipping}, bringing a new
perspective on understanding their role in deep learning. We support our theory
with experiments conducted on a synthetic model and neural networks.
