Classification and Adversarial examples in an Overparameterized Linear
  Model: A Signal Processing Perspective
  State-of-the-art deep learning classifiers are heavily overparameterized with
respect to the amount of training examples and observed to generalize well on
"clean" data, but be highly susceptible to infinitesmal adversarial
perturbations. In this paper, we identify an overparameterized linear ensemble,
that uses the "lifted" Fourier feature map, that demonstrates both of these
behaviors. The input is one-dimensional, and the adversary is only allowed to
perturb these inputs and not the non-linear features directly. We find that the
learned model is susceptible to adversaries in an intermediate regime where
classification generalizes but regression does not. Notably, the susceptibility
arises despite the absence of model mis-specification or label noise, which are
commonly cited reasons for adversarial-susceptibility. These results are
extended theoretically to a random-Fourier-sum setup that exhibits
double-descent behavior. In both feature-setups, the adversarial vulnerability
arises because of a phenomenon we term spatial localization: the predictions of
the learned model are markedly more sensitive in the vicinity of training
points than elsewhere. This sensitivity is a consequence of feature lifting and
is reminiscent of Gibb's and Runge's phenomena from signal processing and
functional analysis. Despite the adversarial susceptibility, we find that
classification with these features can be easier than the more commonly studied
"independent feature" models.
