Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent
  Algorithms via Diffusion Approximation
  Diffusion approximation provides weak approximation for stochastic gradient
descent algorithms in a finite time horizon. In this paper, we introduce new
tools motivated by the backward error analysis of numerical stochastic
differential equations into the theoretical framework of diffusion
approximation, extending the validity of the weak approximation from finite to
infinite time horizon. The new techniques developed in this paper enable us to
characterize the asymptotic behavior of constant-step-size SGD algorithms for
strongly convex objective functions, a goal previously unreachable within the
diffusion approximation framework. Our analysis builds upon a truncated formal
power expansion of the solution of a stochastic modified equation arising from
diffusion approximation, where the main technical ingredient is a
uniform-in-time weak error bound controlling the long-term behavior of the
expansion coefficient functions near the global minimum. We expect these new
techniques to greatly expand the range of applicability of diffusion
approximation to cover wider and deeper aspects of stochastic optimization
algorithms in data science.
