Regularization Effect of Fast Gradient Sign Method and its
  Generalization
  Fast Gradient Sign Method (FGSM) is a popular method to generate adversarial
examples that make neural network models robust against perturbations. Despite
its empirical success, its theoretical property is not well understood. This
paper develops theory to explain the regularization effect of Generalized FGSM,
a class of methods to generate adversarial examples. Motivated from the
relationship between FGSM and LASSO penalty, the asymptotic properties of
Generalized FGSM are derived in the Generalized Linear Model setting, which is
essentially the 1-layer neural network setting with certain activation
functions. In such simple neural network models, I prove that Generalized FGSM
estimation is root n-consistent and weakly oracle under proper conditions. The
asymptotic results are also highly similar to penalized likelihood estimation.
Nevertheless, Generalized FGSM introduces additional bias when data sampling is
not sign neutral, a concept I introduce to describe the balance-ness of the
noise signs. Although the theory in this paper is developed under simple neural
network settings, I argue that it may give insights and justification for FGSM
in deep neural network settings as well.
