Cross-Layer Distillation with Semantic Calibration
  Knowledge distillation is a technique to enhance the generalization ability
of a student model by exploiting outputs from a teacher model. Recently,
feature-map based variants explore knowledge transfer between manually assigned
teacher-student pairs in intermediate layers for further improvement. However,
layer semantics may vary in different neural networks and semantic mismatch in
manual layer associations will lead to performance degeneration due to negative
regularization. To address this issue, we propose Semantic Calibration for
cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper
target layers of the teacher model for each student layer with an attention
mechanism. With a learned attention distribution, each student layer distills
knowledge contained in multiple teacher layers rather than a specific
intermediate layer for appropriate cross-layer supervision. We further provide
theoretical analysis of the association weights and conduct extensive
experiments to demonstrate the effectiveness of our approach. Code is avaliable
at \url{https://github.com/DefangChen/SemCKD}.
