BERT Learns to Teach: Knowledge Distillation with Meta Learning
  We present Knowledge Distillation with Meta Learning (MetaDistil), a simple
yet effective alternative to traditional knowledge distillation (KD) methods
where the teacher model is fixed during training. We show the teacher network
can learn to better transfer knowledge to the student network (i.e., learning
to teach) with the feedback from the performance of the distilled student
network in a meta learning framework. Moreover, we introduce a pilot update
mechanism to improve the alignment between the inner-learner and meta-learner
in meta learning algorithms that focus on an improved inner-learner.
Experiments on various benchmarks show that MetaDistil can yield significant
improvements compared with traditional KD algorithms and is less sensitive to
the choice of different student capacity and hyperparameters, facilitating the
use of KD on different tasks and models.
