PURSUhInT: In Search of Informative Hint Points Based on Layer
  Clustering for Knowledge Distillation
  We propose a novel knowledge distillation methodology for compressing deep
neural networks. One of the most efficient methods for knowledge distillation
is hint distillation, where the student model is injected with information
(hints) from several different layers of the teacher model. Although the
selection of hint points can drastically alter the compression performance,
conventional distillation approaches overlook this fact. Therefore, we propose
a clustering based hint selection methodology, where the layers of teacher
model are clustered with respect to several metrics and the cluster centers are
used as the hint points. Our method is applicable for any student network, once
it is applied on a chosen teacher network. The proposed approach is validated
in CIFAR-100 and ImageNet datasets, using various teacher-student pairs and
numerous hint distillation methods. Our results show that hint points selected
by our algorithm results in superior compression performance with respect to
state-of-the-art knowledge distillation algorithms on the same student models
and datasets.
