Annealing Knowledge Distillation
  Significant memory and computational requirements of large deep neural
networks restrict their application on edge devices. Knowledge distillation
(KD) is a prominent model compression technique for deep neural networks in
which the knowledge of a trained large teacher model is transferred to a
smaller student model. The success of knowledge distillation is mainly
attributed to its training objective function, which exploits the soft-target
information (also known as "dark knowledge") besides the given regular hard
labels in a training set. However, it is shown in the literature that the
larger the gap between the teacher and the student networks, the more difficult
is their training using knowledge distillation. To address this shortcoming, we
propose an improved knowledge distillation method (called Annealing-KD) by
feeding the rich information provided by the teacher's soft-targets
incrementally and more efficiently. Our Annealing-KD technique is based on a
gradual transition over annealed soft-targets generated by the teacher at
different temperatures in an iterative process, and therefore, the student is
trained to follow the annealed teacher output in a step-by-step manner. This
paper includes theoretical and empirical evidence as well as practical
experiments to support the effectiveness of our Annealing-KD method. We did a
comprehensive set of experiments on different tasks such as image
classification (CIFAR-10 and 100) and NLP language inference with BERT-based
models on the GLUE benchmark and consistently got superior results.
