Precise Tradeoffs in Adversarial Training for Linear Regression
  Despite breakthrough performance, modern learning models are known to be
highly vulnerable to small adversarial perturbations in their inputs. While a
wide variety of recent \emph{adversarial training} methods have been effective
at improving robustness to perturbed inputs (robust accuracy), often this
benefit is accompanied by a decrease in accuracy on benign inputs (standard
accuracy), leading to a tradeoff between often competing objectives.
Complicating matters further, recent empirical evidence suggest that a variety
of other factors (size and quality of training data, model size, etc.) affect
this tradeoff in somewhat surprising ways. In this paper we provide a precise
and comprehensive understanding of the role of adversarial training in the
context of linear regression with Gaussian features. In particular, we
characterize the fundamental tradeoff between the accuracies achievable by any
algorithm regardless of computational power or size of the training data.
Furthermore, we precisely characterize the standard/robust accuracy and the
corresponding tradeoff achieved by a contemporary mini-max adversarial training
approach in a high-dimensional regime where the number of data points and the
parameters of the model grow in proportion to each other. Our theory for
adversarial training algorithms also facilitates the rigorous study of how a
variety of factors (size and quality of training data, model
overparametrization etc.) affect the tradeoff between these two competing
accuracies.
