Stochastic modified equations and adaptive stochastic gradient
  algorithms
  We develop the method of stochastic modified equations (SME), in which
stochastic gradient algorithms are approximated in the weak sense by
continuous-time stochastic differential equations. We exploit the continuous
formulation together with optimal control theory to derive novel adaptive
hyper-parameter adjustment policies. Our algorithms have competitive
performance with the added benefit of being robust to varying models and
datasets. This provides a general methodology for the analysis and design of
stochastic gradient algorithms.
