Learning Interpretation with Explainable Knowledge Distillation
  Knowledge Distillation (KD) has been considered as a key solution in model
compression and acceleration in recent years. In KD, a small student model is
generally trained from a large teacher model by minimizing the divergence
between the probabilistic outputs of the two. However, as demonstrated in our
experiments, existing KD methods might not transfer critical explainable
knowledge of the teacher to the student, i.e. the explanations of predictions
made by the two models are not consistent. In this paper, we propose a novel
explainable knowledge distillation model, called XDistillation, through which
both the performance the explanations' information are transferred from the
teacher model to the student model. The XDistillation model leverages the idea
of convolutional autoencoders to approximate the teacher explanations. Our
experiments shows that models trained by XDistillation outperform those trained
by conventional KD methods not only in term of predictive accuracy but also
faithfulness to the teacher models.
