Knowledge Distillation Thrives on Data Augmentation
  Knowledge distillation (KD) is a general deep neural network training
framework that uses a teacher model to guide a student model. Many works have
explored the rationale for its success, however, its interplay with data
augmentation (DA) has not been well recognized so far. In this paper, we are
motivated by an interesting observation in classification: KD loss can benefit
from extended training iterations while the cross-entropy loss does not. We
show this disparity arises because of data augmentation: KD loss can tap into
the extra information from different input views brought by DA. By this
explanation, we propose to enhance KD via a stronger data augmentation scheme
(e.g., mixup, CutMix). Furthermore, an even stronger new DA approach is
developed specifically for KD based on the idea of active learning. The
findings and merits of the proposed method are validated by extensive
experiments on CIFAR-100, Tiny ImageNet, and ImageNet datasets. We can achieve
improved performance simply by using the original KD loss combined with
stronger augmentation schemes, compared to existing state-of-the-art methods,
which employ more advanced distillation losses. In addition, when our
approaches are combined with more advanced distillation losses, we can advance
the state-of-the-art performance even more. On top of the encouraging
performance, this paper also sheds some light on explaining the success of
knowledge distillation. The discovered interplay between KD and DA may inspire
more advanced KD algorithms.
