Adversarial Weight Perturbation Helps Robust Generalization
  The study on improving the robustness of deep neural networks against
adversarial examples grows rapidly in recent years. Among them, adversarial
training is the most promising one, which flattens the input loss landscape
(loss change with respect to input) via training on adversarially perturbed
examples. However, how the widely used weight loss landscape (loss change with
respect to weight) performs in adversarial training is rarely explored. In this
paper, we investigate the weight loss landscape from a new perspective, and
identify a clear correlation between the flatness of weight loss landscape and
robust generalization gap. Several well-recognized adversarial training
improvements, such as early stopping, designing new objective functions, or
leveraging unlabeled data, all implicitly flatten the weight loss landscape.
Based on these observations, we propose a simple yet effective Adversarial
Weight Perturbation (AWP) to explicitly regularize the flatness of weight loss
landscape, forming a double-perturbation mechanism in the adversarial training
framework that adversarially perturbs both inputs and weights. Extensive
experiments demonstrate that AWP indeed brings flatter weight loss landscape
and can be easily incorporated into various existing adversarial training
methods to further boost their adversarial robustness.
