CheXclusion: Fairness gaps in deep chest X-ray classifiers
  Machine learning systems have received much attention recently for their
ability to achieve expert-level performance on clinical tasks, particularly in
medical imaging. Here, we examine the extent to which state-of-the-art deep
learning classifiers trained to yield diagnostic labels from X-ray images are
biased with respect to protected attributes. We train convolution neural
networks to predict 14 diagnostic labels in 3 prominent public chest X-ray
datasets: MIMIC-CXR, Chest-Xray8, CheXpert, as well as a multi-site aggregation
of all those datasets. We evaluate the TPR disparity -- the difference in true
positive rates (TPR) -- among different protected attributes such as patient
sex, age, race, and insurance type as a proxy for socioeconomic status. We
demonstrate that TPR disparities exist in the state-of-the-art classifiers in
all datasets, for all clinical tasks, and all subgroups. A multi-source dataset
corresponds to the smallest disparities, suggesting one way to reduce bias. We
find that TPR disparities are not significantly correlated with a subgroup's
proportional disease burden. As clinical models move from papers to products,
we encourage clinical decision makers to carefully audit for algorithmic
disparities prior to deployment. Our code can be found at,
https://github.com/LalehSeyyed/CheXclusion
