Learn From the Past: Experience Ensemble Knowledge Distillation
  Traditional knowledge distillation transfers "dark knowledge" of a
pre-trained teacher network to a student network, and ignores the knowledge in
the training process of the teacher, which we call teacher's experience.
However, in realistic educational scenarios, learning experience is often more
important than learning results. In this work, we propose a novel knowledge
distillation method by integrating the teacher's experience for knowledge
transfer, named experience ensemble knowledge distillation (EEKD). We save a
moderate number of intermediate models from the training process of the teacher
model uniformly, and then integrate the knowledge of these intermediate models
by ensemble technique. A self-attention module is used to adaptively assign
weights to different intermediate models in the process of knowledge transfer.
Three principles of constructing EEKD on the quality, weights and number of
intermediate models are explored. A surprising conclusion is found that strong
ensemble teachers do not necessarily produce strong students. The experimental
results on CIFAR-100 and ImageNet show that EEKD outperforms the mainstream
knowledge distillation methods and achieves the state-of-the-art. In
particular, EEKD even surpasses the standard ensemble distillation on the
premise of saving training cost.
