Low Curvature Activations Reduce Overfitting in Adversarial Training
  Adversarial training is one of the most effective defenses against
adversarial attacks. Previous works suggest that overfitting is a dominant
phenomenon in adversarial training leading to a large generalization gap
between test and train accuracy in neural networks. In this work, we show that
the observed generalization gap is closely related to the choice of the
activation function. In particular, we show that using activation functions
with low (exact or approximate) curvature values has a regularization effect
that significantly reduces both the standard and robust generalization gaps in
adversarial training. We observe this effect for both differentiable/smooth
activations such as SiLU as well as non-differentiable/non-smooth activations
such as LeakyReLU. In the latter case, the "approximate" curvature of the
activation is low. Finally, we show that for activation functions with low
curvature, the double descent phenomenon for adversarially trained models does
not occur.
