Variational Bayes on Manifolds
  Variational Bayes (VB) has become a widely-used tool for Bayesian inference
in statistics and machine learning. Nonetheless, the development of the
existing VB algorithms is so far generally restricted to the case where the
variational parameter space is Euclidean, which hinders the potential broad
application of VB methods. This paper extends the scope of VB to the case where
the variational parameter space is a Riemannian manifold. We develop an
efficient manifold-based VB algorithm that exploits both the geometric
structure of the constraint parameter space and the information geometry of the
manifold of VB approximating probability distributions. Our algorithm is
provably convergent and achieves a convergence rate of order $\mathcal
O(1/\sqrt{T})$ and $\mathcal O(1/T^{2-2\epsilon})$ for a non-convex evidence
lower bound function and a strongly retraction-convex evidence lower bound
function, respectively. We develop in particular two manifold VB algorithms,
Manifold Gaussian VB and Manifold Neural Net VB, and demonstrate through
numerical experiments that the proposed algorithms are stable, less sensitive
to initialization and compares favourably to existing VB methods.
