Self Regulated Learning Mechanism for Data Efficient Knowledge
  Distillation
  Existing methods for distillation do not efficiently utilize the training
data. This work presents a novel approach to perform distillation using only a
subset of the training data, making it more data-efficient. For this purpose,
the training of the teacher model is modified to include self-regulation
wherein a sample in the training set is used for updating model parameters in
the backward pass either if it is misclassified or the model is not confident
enough in its prediction. This modification restricts the participation of
samples, unlike the conventional training method. The number of times a sample
participates in the self-regulated training process is a measure of its
significance towards the model's knowledge. The significance values are used to
weigh the losses incurred on the corresponding samples in the distillation
process. This method is named significance-based distillation. Two other
methods are proposed for comparison where the student model learns by
distillation and incorporating self-regulation as the teacher model, either
utilizing the significance information computed during the teacher's training
or not. These methods are named hybrid and regulated distillations,
respectively. Experiments on benchmark datasets show that the proposed methods
achieve similar performance as other state-of-the-art methods for knowledge
distillation while utilizing a significantly less number of samples.
