Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights
  Probabilistic neural networks are typically modeled with independent weight
priors, which do not capture weight correlations in the prior and do not
provide a parsimonious interface to express properties in function space. A
desirable class of priors would represent weights compactly, capture
correlations between weights, facilitate calibrated reasoning about
uncertainty, and allow inclusion of prior knowledge about the function space
such as periodicity or dependence on contexts such as inputs. To this end, this
paper introduces two innovations: (i) a Gaussian process-based hierarchical
model for network weights based on unit embeddings that can flexibly encode
correlated weight structures, and (ii) input-dependent versions of these weight
priors that can provide convenient ways to regularize the function space
through the use of kernels defined on contextual inputs. We show these models
provide desirable test-time uncertainty estimates on out-of-distribution data,
demonstrate cases of modeling inductive biases for neural networks with kernels
which help both interpolation and extrapolation from training data, and
demonstrate competitive predictive performance on an active learning benchmark.
