ZeroGrad : Mitigating and Explaining Catastrophic Overfitting in FGSM
  Adversarial Training
  Making deep neural networks robust to small adversarial noises has recently
been sought in many applications. Adversarial training through iterative
projected gradient descent (PGD) has been established as one of the mainstream
ideas to achieve this goal. However, PGD is computationally demanding and often
prohibitive in case of large datasets and models. For this reason, single-step
PGD, also known as FGSM, has recently gained interest in the field.
Unfortunately, FGSM-training leads to a phenomenon called ``catastrophic
overfitting," which is a sudden drop in the adversarial accuracy under the PGD
attack. In this paper, we support the idea that small input gradients play a
key role in this phenomenon, and hence propose to zero the input gradient
elements that are small for crafting FGSM attacks. Our proposed idea, while
being simple and efficient, achieves competitive adversarial accuracy on
various datasets.
