Machine Learning with Multi-Site Imaging Data: An Empirical Study on the
  Impact of Scanner Effects
  This is an empirical study to investigate the impact of scanner effects when
using machine learning on multi-site neuroimaging data. We utilize structural
T1-weighted brain MRI obtained from two different studies, Cam-CAN and UK
Biobank. For the purpose of our investigation, we construct a dataset
consisting of brain scans from 592 age- and sex-matched individuals, 296
subjects from each original study. Our results demonstrate that even after
careful pre-processing with state-of-the-art neuroimaging pipelines a
classifier can easily distinguish between the origin of the data with very high
accuracy. Our analysis on the example application of sex classification
suggests that current approaches to harmonize data are unable to remove
scanner-specific bias leading to overly optimistic performance estimates and
poor generalization. We conclude that multi-site data harmonization remains an
open challenge and particular care needs to be taken when using such data with
advanced machine learning methods for predictive modelling.
