Incorporating Uncertainty in Learning to Defer Algorithms for Safe
  Computer-Aided Diagnosis
  Deep neural networks are increasingly being used for computer-aided
diagnosis, but erroneous diagnoses can be extremely costly for patients. We
propose a learning to defer with uncertainty (LDU) algorithm which identifies
patients for whom diagnostic uncertainty is high and defers them for evaluation
by human experts. LDU was evaluated on the diagnosis of myocardial infarction
(using discharge summaries), the diagnosis of any comorbidities (using
structured data), and the diagnosis of pleural effusion and pneumothorax (using
chest x-rays), and compared with 'learning to defer without uncertainty
information' (LD) and 'direct triage by uncertainty' (DT) methods. LDU achieved
the same F1 score as LD but deferred considerably fewer patients (e.g. 36% vs.
69% deferral rate for diagnosing pleural effusion with an F1 score of 0.96).
Furthermore, even when many patients were assigned the wrong diagnosis with
high confidence (e.g. for the diagnosis of any comorbidities) LDU achieved a
17% increase in F1 score, whereas DT was not applicable. Importantly, the
weight of the defer loss in LDU can be easily adjusted to obtain the desired
trade-off between diagnostic accuracy and deferral rate. In conclusion, LDU can
readily augment any existing diagnostic network to reduce the risk of erroneous
diagnoses in clinical practice.
