CheXpedition: Investigating Generalization Challenges for Translation of
  Chest X-Ray Algorithms to the Clinical Setting
  Although there have been several recent advances in the application of deep
learning algorithms to chest x-ray interpretation, we identify three major
challenges for the translation of chest x-ray algorithms to the clinical
setting. We examine the performance of the top 10 performing models on the
CheXpert challenge leaderboard on three tasks: (1) TB detection, (2) pathology
detection on photos of chest x-rays, and (3) pathology detection on data from
an external institution. First, we find that the top 10 chest x-ray models on
the CheXpert competition achieve an average AUC of 0.851 on the task of
detecting TB on two public TB datasets without fine-tuning or including the TB
labels in training data. Second, we find that the average performance of the
models on photos of x-rays (AUC = 0.916) is similar to their performance on the
original chest x-ray images (AUC = 0.924). Third, we find that the models
tested on an external dataset either perform comparably to or exceed the
average performance of radiologists. We believe that our investigation will
inform rapid translation of deep learning algorithms to safe and effective
clinical decision support tools that can be validated prospectively with large
impact studies and clinical trials.
