Adversarial Training is a Form of Data-dependent Operator Norm
  Regularization
  We establish a theoretical link between adversarial training and operator
norm regularization for deep neural networks. Specifically, we prove that
$\ell_p$-norm constrained projected gradient ascent based adversarial training
with an $\ell_q$-norm loss on the logits of clean and perturbed inputs is
equivalent to data-dependent (p, q) operator norm regularization. This
fundamental connection confirms the long-standing argument that a network's
sensitivity to adversarial examples is tied to its spectral properties and
hints at novel ways to robustify and defend against adversarial attacks. We
provide extensive empirical evidence on state-of-the-art network architectures
to support our theoretical results.
