The curse of overparametrization in adversarial training: Precise
  analysis of robust generalization for random features regression
  Successful deep learning models often involve training neural network
architectures that contain more parameters than the number of training samples.
Such overparametrized models have been extensively studied in recent years, and
the virtues of overparametrization have been established from both the
statistical perspective, via the double-descent phenomenon, and the
computational perspective via the structural properties of the optimization
landscape.
  Despite the remarkable success of deep learning architectures in the
overparametrized regime, it is also well known that these models are highly
vulnerable to small adversarial perturbations in their inputs. Even when
adversarially trained, their performance on perturbed inputs (robust
generalization) is considerably worse than their best attainable performance on
benign inputs (standard generalization). It is thus imperative to understand
how overparametrization fundamentally affects robustness.
  In this paper, we will provide a precise characterization of the role of
overparametrization on robustness by focusing on random features regression
models (two-layer neural networks with random first layer weights). We consider
a regime where the sample size, the input dimension and the number of
parameters grow in proportion to each other, and derive an asymptotically exact
formula for the robust generalization error when the model is adversarially
trained. Our developed theory reveals the nontrivial effect of
overparametrization on robustness and indicates that for adversarially trained
random features models, high overparametrization can hurt robust
generalization.
