Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation:
  A Benchmark Study
  Convolutional neural networks (CNNs) have demonstrated promise in automated
cardiac magnetic resonance imaging segmentation. However, when using CNNs in a
large real world dataset, it is important to quantify segmentation uncertainty
in order to know which segmentations could be problematic. In this work, we
performed a systematic study of Bayesian and non-Bayesian methods for
estimating uncertainty in segmentation neural networks. We evaluated Bayes by
Backprop (BBB), Monte Carlo (MC) Dropout, and Deep Ensembles in terms of
segmentation accuracy, probability calibration, uncertainty on
out-of-distribution images, and segmentation quality control. We tested these
algorithms on datasets with various distortions and observed that Deep
Ensembles outperformed the other methods except for images with heavy noise
distortions. For segmentation quality control, we showed that segmentation
uncertainty is correlated with segmentation accuracy. With the incorporation of
uncertainty estimates, we were able to reduce the percentage of poor
segmentation to 5% by flagging 31% to 48% of the most uncertain images for
manual review, substantially lower than random review of the results without
using neural network uncertainty.
