On the benefits of knowledge distillation for adversarial robustness
  Knowledge distillation is normally used to compress a big network, or
teacher, onto a smaller one, the student, by training it to match its outputs.
Recently, some works have shown that robustness against adversarial attacks can
also be distilled effectively to achieve good rates of robustness on
mobile-friendly models. In this work, however, we take a different point of
view, and show that knowledge distillation can be used directly to boost the
performance of state-of-the-art models in adversarial robustness. In this
sense, we present a thorough analysis and provide general guidelines to distill
knowledge from a robust teacher and boost the clean and adversarial performance
of a student model even further. To that end, we present Adversarial Knowledge
Distillation (AKD), a new framework to improve a model's robust performance,
consisting on adversarially training a student on a mixture of the original
labels and the teacher outputs. Through carefully controlled ablation studies,
we show that using early-stopping, model ensembles and weak adversarial
training are key techniques to maximize performance of the student, and show
that these insights generalize across different robust distillation techniques.
Finally, we provide insights on the effect of robust knowledge distillation on
the dynamics of the student network, and show that AKD mostly improves the
calibration of the network and modify its training dynamics on samples that the
model finds difficult to learn, or even memorize.
