Towards calibrated and scalable uncertainty representations for neural
  networks
  For many applications it is critical to know the uncertainty of a neural
network's predictions. While a variety of neural network parameter estimation
methods have been proposed for uncertainty estimation, they have not been
rigorously compared across uncertainty measures. We assess four of these
parameter estimation methods to calibrate uncertainty estimation using four
different uncertainty measures: entropy, mutual information, aleatoric
uncertainty and epistemic uncertainty. We evaluate the calibration of these
parameter estimation methods using expected calibration error. Additionally, we
propose a novel method of neural network parameter estimation called RECAST,
which combines cosine annealing with warm restarts with Stochastic Gradient
Langevin Dynamics, capturing more diverse parameter distributions. When
benchmarked against mutilated image data, we show that RECAST is
well-calibrated and when combined with predictive entropy and epistemic
uncertainty it offers the best calibrated measure of uncertainty when compared
to recent methods.
