Ensemble Model Patching: A Parameter-Efficient Variational Bayesian
  Neural Network
  Two main obstacles preventing the widespread adoption of variational Bayesian
neural networks are the high parameter overhead that makes them infeasible on
large networks, and the difficulty of implementation, which can be thought of
as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular
because it sidesteps these obstacles. Nevertheless, dropout is often harmful to
model performance when used in networks with batch normalization layers [Li et
al., 2018], which are an indispensable part of modern neural networks. We
construct a general variational family for ensemble-based Bayesian neural
networks that encompasses dropout as a special case. We further present two
specific members of this family that work well with batch normalization layers,
while retaining the benefits of low parameter and programming overhead,
comparable to non-Bayesian training. Our proposed methods improve predictive
accuracy and achieve almost perfect calibration on a ResNet-18 trained with
ImageNet.
