SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
  Adversarial training (AT) has become a popular choice for training robust
networks. However, it tends to sacrifice clean accuracy heavily in favor of
robustness and suffers from a large generalization error. To address these
concerns, we propose Smooth Adversarial Training (SAT), guided by our analysis
on the eigenspectrum of the loss Hessian. We find that curriculum learning, a
scheme that emphasizes on starting "easy" and gradually ramping up on the
"difficulty" of training, smooths the adversarial loss landscape for a suitably
chosen difficulty metric. We present a general formulation for curriculum
learning in the adversarial setting and propose two difficulty metrics based on
the maximal Hessian eigenvalue (H-SAT) and the softmax probability (P-SA). We
demonstrate that SAT stabilizes network training even for a large perturbation
norm and allows the network to operate at a better clean accuracy versus
robustness trade-off curve compared to AT. This leads to a significant
improvement in both clean accuracy and robustness compared to AT, TRADES, and
other baselines. To highlight a few results, our best model improves normal and
robust accuracy by 6% and 1% on CIFAR-100 compared to AT, respectively. On
Imagenette, a ten-class subset of ImageNet, our model outperforms AT by 23% and
3% on normal and robust accuracy respectively.
