A Variational Analysis of Stochastic Gradient Algorithms
  Stochastic Gradient Descent (SGD) is an important algorithm in machine
learning. With constant learning rates, it is a stochastic process that, after
an initial phase of convergence, generates samples from a stationary
distribution. We show that SGD with constant rates can be effectively used as
an approximate posterior inference algorithm for probabilistic modeling.
Specifically, we show how to adjust the tuning parameters of SGD such as to
match the resulting stationary distribution to the posterior. This analysis
rests on interpreting SGD as a continuous-time stochastic process and then
minimizing the Kullback-Leibler divergence between its stationary distribution
and the target posterior. (This is in the spirit of variational inference.) In
more detail, we model SGD as a multivariate Ornstein-Uhlenbeck process and then
use properties of this process to derive the optimal parameters. This
theoretical framework also connects SGD to modern scalable inference
algorithms; we analyze the recently proposed stochastic gradient Fisher scoring
under this perspective. We demonstrate that SGD with properly chosen constant
rates gives a new way to optimize hyperparameters in probabilistic models.
