Direct Uncertainty Prediction for Medical Second Opinions
  The issue of disagreements amongst human experts is a ubiquitous one in both
machine learning and medicine. In medicine, this often corresponds to doctor
disagreements on a patient diagnosis. In this work, we show that machine
learning models can be trained to give uncertainty scores to data instances
that might result in high expert disagreements. In particular, they can
identify patient cases that would benefit most from a medical second opinion.
Our central methodological finding is that Direct Uncertainty Prediction (DUP),
training a model to predict an uncertainty score directly from the raw patient
features, works better than Uncertainty Via Classification, the two-step
process of training a classifier and postprocessing the output distribution to
give an uncertainty score. We show this both with a theoretical result, and on
extensive evaluations on a large scale medical imaging application.
