Faster Stochastic Variational Inference using Proximal-Gradient Methods
  with General Divergence Functions
  Several recent works have explored stochastic gradient methods for
variational inference that exploit the geometry of the variational-parameter
space. However, the theoretical properties of these methods are not
well-understood and these methods typically only apply to
conditionally-conjugate models. We present a new stochastic method for
variational inference which exploits the geometry of the variational-parameter
space and also yields simple closed-form updates even for non-conjugate models.
We also give a convergence-rate analysis of our method and many other previous
methods which exploit the geometry of the space. Our analysis generalizes
existing convergence results for stochastic mirror-descent on non-convex
objectives by using a more general class of divergence functions. Beyond giving
a theoretical justification for a variety of recent methods, our experiments
show that new algorithms derived in this framework lead to state of the art
results on a variety of problems. Further, due to its generality, we expect
that our theoretical analysis could also apply to other applications.
