On Feature Normalization and Data Augmentation
  The moments (a.k.a., mean and standard deviation) of latent features are
often removed as noise when training image recognition models, to increase
stability and reduce training time. However, in the field of image generation,
the moments play a much more central role. Studies have shown that the moments
extracted from instance normalization and positional normalization can roughly
capture style and shape information of an image. Instead of being discarded,
these moments are instrumental to the generation process. In this paper we
propose Moment Exchange, an implicit data augmentation method that encourages
the model to utilize the moment information also for recognition models.
Specifically, we replace the moments of the learned features of one training
image by those of another, and also interpolate the target labels -- forcing
the model to extract training signal from the moments in addition to the
normalized features. As our approach is fast, operates entirely in feature
space, and mixes different signals than prior methods, one can effectively
combine it with existing augmentation approaches. We demonstrate its efficacy
across several recognition benchmark data sets where it improves the
generalization capability of highly competitive baseline networks with
remarkable consistency.
