DAIR: Data Augmented Invariant Regularization
  While deep learning through empirical risk minimization (ERM) has succeeded
at achieving human-level performance at a variety of complex tasks, ERM
generalizes poorly to distribution shift. This is partly explained by
overfitting to spurious features such as background in images or named entities
in natural language. Synthetic data augmentation followed by empirical risk
minimization (DA-ERM) is a simple and widely used solution to remedy this
problem. In addition, consistency regularization could be applied to further
promote model performance to be consistent on the augmented sample and the
original one. In this paper, we propose data augmented invariant regularization
(DAIR), a simple form of consistency regularization that is applied directly on
the loss function rather than intermediate features, making it widely
applicable regardless of network architecture or problem setup. We apply DAIR
to multiple real-world learning problems, namely robust regression, visual
question answering, robust deep neural network training, and neural
task-oriented dialog modeling. Our experiments show that DAIR consistently
outperforms ERM and DA-ERM with little marginal cost and sets new
state-of-the-art results in several benchmarks.
