LTD: Low Temperature Distillation for Robust Adversarial Training
  Adversarial training has been widely used to enhance the robustness of the
neural network models against adversarial attacks. However, there still a
notable gap between the nature accuracy and the robust accuracy. We found one
of the reasons is the commonly used labels, one-hot vectors, hinder the
learning process for image recognition. In this paper, we proposed a method,
called Low Temperature Distillation (LTD), which is based on the knowledge
distillation framework to generate the desired soft labels. Unlike the previous
work, LTD uses relatively low temperature in the teacher model, and employs
different, but fixed, temperatures for the teacher model and the student model.
Moreover, we have investigated the methods to synergize the use of nature data
and adversarial ones in LTD. Experimental results show that without extra
unlabeled data, the proposed method combined with the previous work can achieve
57.72\% and 30.36\% robust accuracy on CIFAR-10 and CIFAR-100 dataset
respectively, which is about 1.21\% improvement of the state-of-the-art methods
in average.
