Projected BNNs: Avoiding weight-space pathologies by learning latent
  representations of neural network weights
  As machine learning systems get widely adopted for high-stake decisions,
quantifying uncertainty over predictions becomes crucial. While modern neural
networks are making remarkable gains in terms of predictive accuracy,
characterizing uncertainty over the parameters of these models is challenging
because of the high dimensionality and complex correlations of the network
parameter space. This paper introduces a novel variational inference framework
for Bayesian neural networks that (1) encodes complex distributions in
high-dimensional parameter space with representations in a low-dimensional
latent space, and (2) performs inference efficiently on the low-dimensional
representations. Across a large array of synthetic and real-world datasets, we
show that our method improves uncertainty characterization and model
generalization when compared with methods that work directly in the parameter
space.
