Objective Evaluation of Deep Uncertainty Predictions for COVID-19
  Detection
  Deep neural networks (DNNs) have been widely applied for detecting COVID-19
in medical images. Existing studies mainly apply transfer learning and other
data representation strategies to generate accurate point estimates. The
generalization power of these networks is always questionable due to being
developed using small datasets and failing to report their predictive
confidence. Quantifying uncertainties associated with DNN predictions is a
prerequisite for their trusted deployment in medical settings. Here we apply
and evaluate three uncertainty quantification techniques for COVID-19 detection
using chest X-Ray (CXR) images. The novel concept of uncertainty confusion
matrix is proposed and new performance metrics for the objective evaluation of
uncertainty estimates are introduced. Through comprehensive experiments, it is
shown that networks pertained on CXR images outperform networks pretrained on
natural image datasets such as ImageNet. Qualitatively and quantitatively
evaluations also reveal that the predictive uncertainty estimates are
statistically higher for erroneous predictions than correct predictions.
Accordingly, uncertainty quantification methods are capable of flagging risky
predictions with high uncertainty estimates. We also observe that ensemble
methods more reliably capture uncertainties during the inference.
