Exploring the Uncertainty Properties of Neural Networks' Implicit Priors
  in the Infinite-Width Limit
  Modern deep learning models have achieved great success in predictive
accuracy for many data modalities. However, their application to many
real-world tasks is restricted by poor uncertainty estimates, such as
overconfidence on out-of-distribution (OOD) data and ungraceful failing under
distributional shift. Previous benchmarks have found that ensembles of neural
networks (NNs) are typically the best calibrated models on OOD data. Inspired
by this, we leverage recent theoretical advances that characterize the
function-space prior of an ensemble of infinitely-wide NNs as a Gaussian
process, termed the neural network Gaussian process (NNGP). We use the NNGP
with a softmax link function to build a probabilistic model for multi-class
classification and marginalize over the latent Gaussian outputs to sample from
the posterior. This gives us a better understanding of the implicit prior NNs
place on function space and allows a direct comparison of the calibration of
the NNGP and its finite-width analogue. We also examine the calibration of
previous approaches to classification with the NNGP, which treat classification
problems as regression to the one-hot labels. In this case the Bayesian
posterior is exact, and we compare several heuristics to generate a categorical
distribution over classes. We find these methods are well calibrated under
distributional shift. Finally, we consider an infinite-width final layer in
conjunction with a pre-trained embedding. This replicates the important
practical use case of transfer learning and allows scaling to significantly
larger datasets. As well as achieving competitive predictive accuracy, this
approach is better calibrated than its finite width analogue.
