Drawing Multiple Augmentation Samples Per Image During Training
  Efficiently Decreases Test Error
  In computer vision, it is standard practice to draw a single sample from the
data augmentation procedure for each unique image in the mini-batch. However
recent work has suggested drawing multiple samples can achieve higher test
accuracies. In this work, we provide a detailed empirical evaluation of how the
number of augmentation samples per unique image influences model performance on
held out data when training deep ResNets. We demonstrate drawing multiple
samples per image consistently enhances the test accuracy achieved for both
small and large batch training. Crucially, this benefit arises even if
different numbers of augmentations per image perform the same number of
parameter updates and gradient evaluations (requiring the same total compute).
Although prior work has found variance in the gradient estimate arising from
subsampling the dataset has an implicit regularization benefit, our experiments
suggest variance which arises from the data augmentation process harms
generalization. We apply these insights to the highly performant NFNet-F5,
achieving 86.8$\%$ top-1 w/o extra data on ImageNet.
