Investigating underdiagnosis of AI algorithms in the presence of
  multiple sources of dataset bias
  Deep learning models have shown great potential for image-based diagnosis
assisting clinical decision making. At the same time, an increasing number of
reports raise concerns about the potential risk that machine learning could
amplify existing health disparities due to human biases that are embedded in
the training data. It is of great importance to carefully investigate the
extent to which biases may be reproduced or even amplified if we wish to build
fair artificial intelligence systems. Seyyed-Kalantari et al. advance this
conversation by analysing the performance of a disease classifier across
population subgroups. They raise performance disparities related to
underdiagnosis as a point of concern; we identify areas from this analysis
which we believe deserve additional attention. Specifically, we wish to
highlight some theoretical and practical difficulties associated with assessing
model fairness through testing on data drawn from the same biased distribution
as the training data, especially when the sources and amount of biases are
unknown.
