Was that so hard? Estimating human classification difficulty
  When doctors are trained to diagnose a specific disease, they learn faster
when presented with cases in order of increasing difficulty. This creates the
need for automatically estimating how difficult it is for doctors to classify a
given case. In this paper, we introduce methods for estimating how hard it is
for a doctor to diagnose a case represented by a medical image, both when
ground truth difficulties are available for training, and when they are not.
Our methods are based on embeddings obtained with deep metric learning.
Additionally, we introduce a practical method for obtaining ground truth human
difficulty for each image case in a dataset using self-assessed certainty. We
apply our methods to two different medical datasets, achieving high Kendall
rank correlation coefficients, showing that we outperform existing methods by a
large margin on our problem and data.
