CheXbreak: Misclassification Identification for Deep Learning Models
  Interpreting Chest X-rays
  A major obstacle to the integration of deep learning models for chest x-ray
interpretation into clinical settings is the lack of understanding of their
failure modes. In this work, we first investigate whether there are patient
subgroups that chest x-ray models are likely to misclassify. We find that
patient age and the radiographic finding of lung lesion, pneumothorax or
support devices are statistically relevant features for predicting
misclassification for some chest x-ray models. Second, we develop
misclassification predictors on chest x-ray models using their outputs and
clinical features. We find that our best performing misclassification
identifier achieves an AUROC close to 0.9 for most diseases. Third, employing
our misclassification identifiers, we develop a corrective algorithm to
selectively flip model predictions that have high likelihood of
misclassification at inference time. We observe F1 improvement on the
prediction of Consolidation (0.008 [95% CI 0.005, 0.010]) and Edema (0.003,
[95% CI 0.001, 0.006]). By carrying out our investigation on ten distinct and
high-performing chest x-ray models, we are able to derive insights across model
architectures and offer a generalizable framework applicable to other medical
imaging tasks.
