Automated Model Selection with Bayesian Quadrature
  We present a novel technique for tailoring Bayesian quadrature (BQ) to model
selection. The state-of-the-art for comparing the evidence of multiple models
relies on Monte Carlo methods, which converge slowly and are unreliable for
computationally expensive models. Previous research has shown that BQ offers
sample efficiency superior to Monte Carlo in computing the evidence of an
individual model. However, applying BQ directly to model comparison may waste
computation producing an overly-accurate estimate for the evidence of a clearly
poor model. We propose an automated and efficient algorithm for computing the
most-relevant quantity for model selection: the posterior probability of a
model. Our technique maximizes the mutual information between this quantity and
observations of the models' likelihoods, yielding efficient acquisition of
samples across disparate model spaces when likelihood observations are limited.
Our method produces more-accurate model posterior estimates using fewer model
likelihood evaluations than standard Bayesian quadrature and Monte Carlo
estimators, as we demonstrate on synthetic and real-world examples.
