Semi-Implicit Variational Inference
  Semi-implicit variational inference (SIVI) is introduced to expand the
commonly used analytic variational distribution family, by mixing the
variational parameter with a flexible distribution. This mixing distribution
can assume any density function, explicit or not, as long as independent random
samples can be generated via reparameterization. Not only does SIVI expand the
variational family to incorporate highly flexible variational distributions,
including implicit ones that have no analytic density functions, but also
sandwiches the evidence lower bound (ELBO) between a lower bound and an upper
bound, and further derives an asymptotically exact surrogate ELBO that is
amenable to optimization via stochastic gradient ascent. With a substantially
expanded variational family and a novel optimization algorithm, SIVI is shown
to closely match the accuracy of MCMC in inferring the posterior in a variety
of Bayesian inference tasks.
