Reliable Adversarial Distillation with Unreliable Teachers
  In ordinary distillation, student networks are trained with soft labels (SLs)
given by pretrained teacher networks, and students are expected to improve upon
teachers since SLs are stronger supervision than the original hard labels.
However, when considering adversarial robustness, teachers may become
unreliable and adversarial distillation may not work: teachers are pretrained
on their own adversarial data, and it is too demanding to require that teachers
are also good at every adversarial data queried by students. Therefore, in this
paper, we propose reliable introspective adversarial distillation (IAD) where
students partially instead of fully trust their teachers. Specifically, IAD
distinguishes between three cases given a query of a natural data (ND) and the
corresponding adversarial data (AD): (a) if a teacher is good at AD, its SL is
fully trusted; (b) if a teacher is good at ND but not AD, its SL is partially
trusted and the student also takes its own SL into account; (c) otherwise, the
student only relies on its own SL. Experiments demonstrate the effectiveness of
IAD for improving upon teachers in terms of adversarial robustness.
