On the Impact of Hard Adversarial Instances on Overfitting in
  Adversarial Training
  Adversarial training is a popular method to robustify models against
adversarial attacks. However, it exhibits much more severe overfitting than
training on clean inputs. In this work, we investigate this phenomenon from the
perspective of training instances, i.e., training input-target pairs. Based on
a quantitative metric measuring instances' difficulty, we analyze the model's
behavior on training instances of different difficulty levels. This lets us
show that the decay in generalization performance of adversarial training is a
result of the model's attempt to fit hard adversarial instances. We
theoretically verify our observations for both linear and general nonlinear
models, proving that models trained on hard instances have worse generalization
performance than ones trained on easy instances. Furthermore, we prove that the
difference in the generalization gap between models trained by instances of
different difficulty levels increases with the size of the adversarial budget.
Finally, we conduct case studies on methods mitigating adversarial overfitting
in several scenarios. Our analysis shows that methods successfully mitigating
adversarial overfitting all avoid fitting hard adversarial instances, while
ones fitting hard adversarial instances do not achieve true robustness.
