On Stein Variational Neural Network Ensembles
  Ensembles of deep neural networks have achieved great success recently, but
they do not offer a proper Bayesian justification. Moreover, while they allow
for averaging of predictions over several hypotheses, they do not provide any
guarantees for their diversity, leading to redundant solutions in function
space. In contrast, particle-based inference methods, such as Stein variational
gradient descent (SVGD), offer a Bayesian framework, but rely on the choice of
a kernel to measure the similarity between ensemble members. In this work, we
study different SVGD methods operating in the weight space, function space, and
in a hybrid setting. We compare the SVGD approaches to other ensembling-based
methods in terms of their theoretical properties and assess their empirical
performance on synthetic and real-world tasks. We find that SVGD using
functional and hybrid kernels can overcome the limitations of deep ensembles.
It improves on functional diversity and uncertainty estimation and approaches
the true Bayesian posterior more closely. Moreover, we show that using
stochastic SVGD updates, as opposed to the standard deterministic ones, can
further improve the performance.
