What Do Adversarially trained Neural Networks Focus: A Fourier
  Domain-based Study
  Although many fields have witnessed the superior performance brought about by
deep learning, the robustness of neural networks remains an open issue.
Specifically, a small adversarial perturbation on the input may cause the model
to produce a completely different output. Such poor robustness implies many
potential hazards, especially in security-critical applications, e.g.,
autonomous driving and mobile robotics. This work studies what information the
adversarially trained model focuses on. Empirically, we notice that the
differences between the clean and adversarial data are mainly distributed in
the low-frequency region. We then find that an adversarially-trained model is
more robust than its naturally-trained counterpart due to the reason that the
former pays more attention to learning the dominant information in
low-frequency components. In addition, we consider two common ways to improve
model robustness, namely, by data augmentation and by using stronger network
architectures, and understand these techniques from a frequency-domain
perspective. We are hopeful this work can shed light on the design of more
robust neural networks.
