The FMRIB Variational Bayesian Inference Tutorial II: Stochastic
  Variational Bayes
  Bayesian methods have proved powerful in many applications for the inference
of model parameters from data. These methods are based on Bayes' theorem, which
itself is deceptively simple. However, in practice the computations required
are intractable even for simple cases. Hence methods for Bayesian inference
have historically either been significantly approximate, e.g., the Laplace
approximation, or achieve samples from the exact solution at significant
computational expense, e.g., Markov Chain Monte Carlo methods. Since around the
year 2000 so-called Variational approaches to Bayesian inference have been
increasingly deployed. In its most general form Variational Bayes (VB) involves
approximating the true posterior probability distribution via another more
'manageable' distribution, the aim being to achieve as good an approximation as
possible. In the original FMRIB Variational Bayes tutorial we documented an
approach to VB based that took a 'mean field' approach to forming the
approximate posterior, required the conjugacy of prior and likelihood, and
exploited the Calculus of Variations, to derive an iterative series of update
equations, akin to Expectation Maximisation. In this tutorial we revisit VB,
but now take a stochastic approach to the problem that potentially circumvents
some of the limitations imposed by the earlier methodology. This new approach
bears a lot of similarity to, and has benefited from, computational methods
applied to machine learning algorithms. Although, what we document here is
still recognisably Bayesian inference in the classic sense, and not an attempt
to use machine learning as a black-box to solve the inference problem.
