A Benchmark of Medical Out of Distribution Detection
  Motivation: Deep learning models deployed for use on medical tasks can be
equipped with Out-of-Distribution Detection (OoDD) methods in order to avoid
erroneous predictions. However it is unclear which OoDD method should be used
in practice. Specific Problem: Systems trained for one particular domain of
images cannot be expected to perform accurately on images of a different
domain. These images should be flagged by an OoDD method prior to diagnosis.
Our approach: This paper defines 3 categories of OoD examples and benchmarks
popular OoDD methods in three domains of medical imaging: chest X-ray, fundus
imaging, and histology slides. Results: Our experiments show that despite
methods yielding good results on some categories of out-of-distribution
samples, they fail to recognize images close to the training distribution.
Conclusion: We find a simple binary classifier on the feature representation
has the best accuracy and AUPRC on average. Users of diagnostic tools which
employ these OoDD methods should still remain vigilant that images very close
to the training distribution yet not in it could yield unexpected results.
