Multiplicative noise and heavy tails in stochastic optimization
  Although stochastic optimization is central to modern machine learning, the
precise mechanisms underlying its success, and in particular, the precise role
of the stochasticity, still remain unclear. Modelling stochastic optimization
algorithms as discrete random recurrence relations, we show that multiplicative
noise, as it commonly arises due to variance in local rates of convergence,
results in heavy-tailed stationary behaviour in the parameters. A detailed
analysis is conducted for SGD applied to a simple linear regression problem,
followed by theoretical results for a much larger class of models (including
non-linear and non-convex) and optimizers (including momentum, Adam, and
stochastic Newton), demonstrating that our qualitative results hold much more
generally. In each case, we describe dependence on key factors, including step
size, batch size, and data variability, all of which exhibit similar
qualitative behavior to recent empirical results on state-of-the-art neural
network models from computer vision and natural language processing.
Furthermore, we empirically demonstrate how multiplicative noise and
heavy-tailed structure improve capacity for basin hopping and exploration of
non-convex loss surfaces, over commonly-considered stochastic dynamics with
only additive noise and light-tailed structure.
