Wide Bayesian neural networks have a simple weight posterior: theory and accelerated sampling
We introduce repriorisation, a data-dependent reparameterisation which
transforms a Bayesian neural network (BNN) posterior to a distribution whose KL
divergence to the BNN prior vanishes as layer widths grow. The repriorisation
map acts directly on parameters, and its analytic simplicity complements the
known neural network Gaussian process (NNGP) behaviour of wide BNNs in function
space. Exploiting the repriorisation, we develop a Markov chain Monte Carlo
(MCMC) posterior sampling algorithm which mixes faster the wider the BNN. This
contrasts with the typically poor performance of MCMC in high dimensions. We
observe up to 50x higher effective sample size relative to no reparametrisation
for both fully-connected and residual networks. Improvements are achieved at
all widths, with the margin between reparametrised and standard BNNs growing
with layer width.