Continuous-time Models for Stochastic Optimization Algorithms
  We propose new continuous-time formulations for first-order stochastic
optimization algorithms such as mini-batch gradient descent and
variance-reduced methods. We exploit these continuous-time models, together
with simple Lyapunov analysis as well as tools from stochastic calculus, in
order to derive convergence bounds for various types of non-convex functions.
Guided by such analysis, we show that the same Lyapunov arguments hold in
discrete-time, leading to matching rates. In addition, we use these models and
Ito calculus to infer novel insights on the dynamics of SGD, proving that a
decreasing learning rate acts as time warping or, equivalently, as landscape
stretching.
