Beyond Marginal Uncertainty: How Accurately can Bayesian Regression
  Models Estimate Posterior Predictive Correlations?
  While uncertainty estimation is a well-studied topic in deep learning, most
such work focuses on marginal uncertainty estimates, i.e. the predictive mean
and variance at individual input locations. But it is often more useful to
estimate predictive correlations between the function values at different input
locations. In this paper, we consider the problem of benchmarking how
accurately Bayesian models can estimate predictive correlations. We first
consider a downstream task which depends on posterior predictive correlations:
transductive active learning (TAL). We find that TAL makes better use of
models' uncertainty estimates than ordinary active learning, and recommend this
as a benchmark for evaluating Bayesian models. Since TAL is too expensive and
indirect to guide development of algorithms, we introduce two metrics which
more directly evaluate the predictive correlations and which can be computed
efficiently: meta-correlations (i.e. the correlations between the models
correlation estimates and the true values), and cross-normalized likelihoods
(XLL). We validate these metrics by demonstrating their consistency with TAL
performance and obtain insights about the relative performance of current
Bayesian neural net and Gaussian process models.
