Mean-Field Approximation to Gaussian-Softmax Integral with Application
  to Uncertainty Estimation
  Many methods have been proposed to quantify the predictive uncertainty
associated with the outputs of deep neural networks. Among them, ensemble
methods often lead to state-of-the-art results, though they require
modifications to the training procedures and are computationally costly for
both training and inference. In this paper, we propose a new single-model based
approach. The main idea is inspired by the observation that we can "simulate"
an ensemble of models by drawing from a Gaussian distribution, with a form
similar to those from the asymptotic normality theory, infinitesimal Jackknife,
Laplacian approximation to Bayesian neural networks, and trajectories in
stochastic gradient descents. However, instead of using each model in the
"ensemble" to predict and then aggregating their predictions, we integrate the
Gaussian distribution and the softmax outputs of the neural networks. We use a
mean-field approximation formula to compute this analytically intractable
integral. The proposed approach has several appealing properties: it functions
as an ensemble without requiring multiple models, and it enables closed-form
approximate inference using only the first and second moments of the Gaussian.
Empirically, the proposed approach performs competitively when compared to
state-of-the-art methods, including deep ensembles, temperature scaling,
dropout and Bayesian NNs, on standard uncertainty estimation tasks. It also
outperforms many methods on out-of-distribution detection.
