Data Augmentation as Feature Manipulation: a story of desert cows and
  grass cows
  Data augmentation is a cornerstone of the machine learning pipeline, yet its
theoretical underpinnings remain unclear. Is it merely a way to artificially
augment the data set size? Or is it about encouraging the model to satisfy
certain invariance? In this work we consider another angle, and we study the
effect of data augmentation on the dynamic of the learning process. We find
that data augmentation can alter the relative importance of various features,
effectively making certain informative but hard to learn features more likely
to be captured in the learning process. Importantly, we show that this effect
is more pronounced for non-linear models, such as neural networks. Our main
contribution is a detailed analysis of data augmentation on the learning
dynamic for a two layer convolutional neural network in the recently proposed
multi-view model by Allen-Zhu and Li [2020]. We complement this analysis with
further experimental evidence that data augmentation can be viewed as a form of
feature manipulation.
