"Name that manufacturer". Relating image acquisition bias with task
  complexity when training deep learning models: experiments on head CT
  As interest in applying machine learning techniques for medical images
continues to grow at a rapid pace, models are starting to be developed and
deployed for clinical applications. In the clinical AI model development
lifecycle (described by Lu et al. [1]), a crucial phase for machine learning
scientists and clinicians is the proper design and collection of the data
cohort. The ability to recognize various forms of biases and distribution
shifts in the dataset is critical at this step. While it remains difficult to
account for all potential sources of bias, techniques can be developed to
identify specific types of bias in order to mitigate their impact. In this work
we analyze how the distribution of scanner manufacturers in a dataset can
contribute to the overall bias of deep learning models. We evaluate
convolutional neural networks (CNN) for both classification and segmentation
tasks, specifically two state-of-the-art models: ResNet [2] for classification
and U-Net [3] for segmentation. We demonstrate that CNNs can learn to
distinguish the imaging scanner manufacturer and that this bias can
substantially impact model performance for both classification and segmentation
tasks. By creating an original synthesis dataset of brain data mimicking the
presence of more or less subtle lesions we also show that this bias is related
to the difficulty of the task. Recognition of such bias is critical to develop
robust, generalizable models that will be crucial for clinical applications in
real-world data distributions.
