Learning to Teach with Student Feedback
  Knowledge distillation (KD) has gained much attention due to its
effectiveness in compressing large-scale pre-trained models. In typical KD
methods, the small student model is trained to match the soft targets generated
by the big teacher model. However, the interaction between student and teacher
is one-way. The teacher is usually fixed once trained, resulting in static soft
targets to be distilled. This one-way interaction leads to the teacher's
inability to perceive the characteristics of the student and its training
progress. To address this issue, we propose Interactive Knowledge Distillation
(IKD), which also allows the teacher to learn to teach from the feedback of the
student. In particular, IKD trains the teacher model to generate specific soft
target at each training step for a certain student. Joint optimization for both
teacher and student is achieved by two iterative steps: a course step to
optimize student with the soft target of teacher, and an exam step to optimize
teacher with the feedback of student. IKD is a general framework that is
orthogonal to most existing knowledge distillation methods. Experimental
results show that IKD outperforms traditional KD methods on various NLP tasks.
