Knowledge Distillation in Deep Learning and its Applications
  Deep learning based models are relatively large, and it is hard to deploy
such models on resource-limited devices such as mobile phones and embedded
devices. One possible solution is knowledge distillation whereby a smaller
model (student model) is trained by utilizing the information from a larger
model (teacher model). In this paper, we present a survey of knowledge
distillation techniques applied to deep learning models. To compare the
performances of different techniques, we propose a new metric called
distillation metric. Distillation metric compares different knowledge
distillation algorithms based on sizes and accuracy scores. Based on the
survey, some interesting conclusions are drawn and presented in this paper.
