Quantifying the probable approximation error of probabilistic inference
  programs
  This paper introduces a new technique for quantifying the approximation error
of a broad class of probabilistic inference programs, including ones based on
both variational and Monte Carlo approaches. The key idea is to derive a
subjective bound on the symmetrized KL divergence between the distribution
achieved by an approximate inference program and its true target distribution.
The bound's validity (and subjectivity) rests on the accuracy of two auxiliary
probabilistic programs: (i) a "reference" inference program that defines a gold
standard of accuracy and (ii) a "meta-inference" program that answers the
question "what internal random choices did the original approximate inference
program probably make given that it produced a particular result?" The paper
includes empirical results on inference problems drawn from linear regression,
Dirichlet process mixture modeling, HMMs, and Bayesian networks. The
experiments show that the technique is robust to the quality of the reference
inference program and that it can detect implementation bugs that are not
apparent from predictive performance.
