Deeply-Supervised Knowledge Distillation
  Knowledge distillation aims to enhance the performance of a lightweight
student model by exploiting the knowledge from a pre-trained cumbersome teacher
model. However, in the traditional knowledge distillation, teacher predictions
are only used to provide the supervisory signal for the last layer of the
student model, which may result in those shallow student layers lacking
accurate training guidance in the layer-by-layer back propagation and thus
hinders effective knowledge transfer. To address this issue, we propose
Deeply-Supervised Knowledge Distillation (DSKD), which fully utilizes class
predictions and feature maps of the teacher model to supervise the training of
shallow student layers. A loss-based weight allocation strategy is developed in
DSKD to adaptively balance the learning process of each shallow layer, so as to
further improve the student performance. Extensive experiments show that the
performance of DSKD consistently exceeds state-of-the-art methods on various
teacher-student models, confirming the effectiveness of our proposed method.
