Collaborative Teacher-Student Learning via Multiple Knowledge Transfer
  Knowledge distillation (KD), as an efficient and effective model compression
technique, has been receiving considerable attention in deep learning. The key
to its success is to transfer knowledge from a large teacher network to a small
student one. However, most of the existing knowledge distillation methods
consider only one type of knowledge learned from either instance features or
instance relations via a specific distillation strategy in teacher-student
learning. There are few works that explore the idea of transferring different
types of knowledge with different distillation strategies in a unified
framework. Moreover, the frequently used offline distillation suffers from a
limited learning capacity due to the fixed teacher-student architecture. In
this paper we propose a collaborative teacher-student learning via multiple
knowledge transfer (CTSL-MKT) that prompts both self-learning and collaborative
learning. It allows multiple students learn knowledge from both individual
instances and instance relations in a collaborative way. While learning from
themselves with self-distillation, they can also guide each other via online
distillation. The experiments and ablation studies on four image datasets
demonstrate that the proposed CTSL-MKT significantly outperforms the
state-of-the-art KD methods.
