On the Generalization Properties of Adversarial Training
  Modern machine learning and deep learning models are shown to be vulnerable
when testing data are slightly perturbed. Existing theoretical studies of
adversarial training algorithms mostly focus on either adversarial training
losses or local convergence properties. In contrast, this paper studies the
generalization performance of a generic adversarial training algorithm.
Specifically, we consider linear regression models and two-layer neural
networks (with lazy training) using squared loss under low-dimensional and
high-dimensional regimes. In the former regime, after overcoming the
non-smoothness of adversarial training, the adversarial risk of the trained
models can converge to the minimal adversarial risk. In the latter regime, we
discover that data interpolation prevents the adversarially robust estimator
from being consistent. Therefore, inspired by successes of the least absolute
shrinkage and selection operator (LASSO), we incorporate the L1 penalty in the
high dimensional adversarial learning and show that it leads to consistent
adversarially robust estimation. A series of numerical studies are conducted to
demonstrate how the smoothness and L1 penalization help improve the adversarial
robustness of DNN models.
