On Calibrated Model Uncertainty in Deep Learning
  Estimated uncertainty by approximate posteriors in Bayesian neural networks
are prone to miscalibration, which leads to overconfident predictions in
critical tasks that have a clear asymmetric cost or significant losses. Here,
we extend the approximate inference for the loss-calibrated Bayesian framework
to dropweights based Bayesian neural networks by maximising expected utility
over a model posterior to calibrate uncertainty in deep learning. Furthermore,
we show that decisions informed by loss-calibrated uncertainty can improve
diagnostic performance to a greater extent than straightforward alternatives.
We propose Maximum Uncertainty Calibration Error (MUCE) as a metric to measure
calibrated confidence, in addition to its prediction especially for high-risk
applications, where the goal is to minimise the worst-case deviation between
error and estimated uncertainty. In experiments, we show the correlation
between error in prediction and estimated uncertainty by interpreting
Wasserstein distance as the accuracy of prediction. We evaluated the
effectiveness of our approach to detecting Covid-19 from X-Ray images.
Experimental results show that our method reduces miscalibration considerably,
without impacting the models accuracy and improves reliability of
computer-based diagnostics.
