Variational Inference as Iterative Projection in a Bayesian Hilbert
  Space
  Variational Bayesian inference is an important machine-learning tool that
finds application from statistics to robotics. The goal is to find an
approximate probability density function (PDF) from a chosen family that is in
some sense 'closest' to the full Bayesian posterior. Closeness is typically
defined through the selection of an appropriate loss functional such as the
Kullback-Leibler (KL) divergence. In this paper, we explore a new formulation
of variational inference by exploiting the fact that (most) PDFs are members of
a Bayesian Hilbert space under careful definitions of vector addition, scalar
multiplication and an inner product. We show that variational inference based
on KL divergence then amounts to an iterative projection, in the Euclidean
sense, of the Bayesian posterior onto a subspace corresponding to the selected
approximation family. We work through the details of this general framework for
the specific case of the Gaussian approximation family and show the equivalence
to another Gaussian variational inference approach. We furthermore discuss the
implications for systems that exhibit sparsity, which is handled naturally in
Bayesian space, and give an example of a high-dimensional robotic state
estimation problem that can be handled as a result. Finally, we provide some
preliminary examples of how the approach could be applied to non-Gaussian
inference.
