CheXternal: Generalization of Deep Learning Models for Chest X-ray
  Interpretation to Photos of Chest X-rays and External Clinical Settings
  Recent advances in training deep learning models have demonstrated the
potential to provide accurate chest X-ray interpretation and increase access to
radiology expertise. However, poor generalization due to data distribution
shifts in clinical settings is a key barrier to implementation. In this study,
we measured the diagnostic performance for 8 different chest X-ray models when
applied to (1) smartphone photos of chest X-rays and (2) external datasets
without any finetuning. All models were developed by different groups and
submitted to the CheXpert challenge, and re-applied to test datasets without
further tuning. We found that (1) on photos of chest X-rays, all 8 models
experienced a statistically significant drop in task performance, but only 3
performed significantly worse than radiologists on average, and (2) on the
external set, none of the models performed statistically significantly worse
than radiologists, and five models performed statistically significantly better
than radiologists. Our results demonstrate that some chest X-ray models, under
clinically relevant distribution shifts, were comparable to radiologists while
other models were not. Future work should investigate aspects of model training
procedures and dataset collection that influence generalization in the presence
of data distribution shifts.
