Teacher-Class Network: A Neural Network Compression Mechanism
  To reduce the overwhelming size of Deep Neural Networks (DNN) teacher-student
methodology tries to transfer knowledge from a complex teacher network to a
simple student network. We instead propose a novel method called the
teacher-class network consisting of a single teacher and multiple student
networks (i.e. class of students). Instead of transferring knowledge to one
student only, the proposed method transfers a chunk of knowledge to each
student. Our students are not trained for problem-specific logits, they are
trained to mimic knowledge (dense representation) learned by the teacher
network thus the combined knowledge learned by the class of students can be
used to solve other problems as well. The proposed teacher-class architecture
is evaluated on several benchmark datasets such as MNIST, Fashion MNIST, IMDB
Movie Reviews, CAMVid, CIFAR-10 and ImageNet on multiple tasks including image
classification, sentiment classification and segmentation. Our approach
outperforms the state of-the-art single student approach in terms of accuracy
as well as computational cost while achieving 10-30 times reduction in
parameters.
