On the Effectiveness of Mode Exploration in Bayesian Model Averaging for
  Neural Networks
  Multiple techniques for producing calibrated predictive probabilities using
deep neural networks in supervised learning settings have emerged that leverage
approaches to ensemble diverse solutions discovered during cyclic training or
training from multiple random starting points (deep ensembles). However, only a
limited amount of work has investigated the utility of exploring the local
region around each diverse solution (posterior mode). Using three well-known
deep architectures on the CIFAR-10 dataset, we evaluate several simple methods
for exploring local regions of the weight space with respect to Brier score,
accuracy, and expected calibration error. We consider both Bayesian inference
techniques (variational inference and Hamiltonian Monte Carlo applied to the
softmax output layer) as well as utilizing the stochastic gradient descent
trajectory near optima. While adding separate modes to the ensemble uniformly
improves performance, we show that the simple mode exploration methods
considered here produce little to no improvement over ensembles without mode
exploration.
