Regularising for invariance to data augmentation improves supervised
  learning
  Data augmentation is used in machine learning to make the classifier
invariant to label-preserving transformations. Usually this invariance is only
encouraged implicitly by including a single augmented input during training.
However, several works have recently shown that using multiple augmentations
per input can improve generalisation or can be used to incorporate invariances
more explicitly. In this work, we first empirically compare these recently
proposed objectives that differ in whether they rely on explicit or implicit
regularisation and at what level of the predictor they encode the invariances.
We show that the predictions of the best performing method are also the most
similar when compared on different augmentations of the same input. Inspired by
this observation, we propose an explicit regulariser that encourages this
invariance on the level of individual model predictions. Through extensive
experiments on CIFAR-100 and ImageNet we show that this explicit regulariser
(i) improves generalisation and (ii) equalises performance differences between
all considered objectives. Our results suggest that objectives that encourage
invariance on the level of the neural network itself generalise better than
those that achieve invariance by averaging predictions of non-invariant models.
