Individual predictions matter: Assessing the effect of data ordering in
  training fine-tuned CNNs for medical imaging
  We reproduced the results of CheXNet with fixed hyperparameters and 50
different random seeds to identify 14 finding in chest radiographs (x-rays).
Because CheXNet fine-tunes a pre-trained DenseNet, the random seed affects the
ordering of the batches of training data but not the initialized model weights.
We found substantial variability in predictions for the same radiograph across
model runs (mean ln[(maximum probability)/(minimum probability)] 2.45,
coefficient of variation 0.543). This individual radiograph-level variability
was not fully reflected in the variability of AUC on a large test set.
Averaging predictions from 10 models reduced variability by nearly 70% (mean
coefficient of variation from 0.543 to 0.169, t-test 15.96, p-value < 0.0001).
We encourage researchers to be aware of the potential variability of CNNs and
ensemble predictions from multiple models to minimize the effect this
variability may have on the care of individual patients when these models are
deployed clinically.
