Role-Wise Data Augmentation for Knowledge Distillation
  Knowledge Distillation (KD) is a common method for transferring the
``knowledge'' learned by one machine learning model (the \textit{teacher}) into
another model (the \textit{student}), where typically, the teacher has a
greater capacity (e.g., more parameters or higher bit-widths). To our
knowledge, existing methods overlook the fact that although the student absorbs
extra knowledge from the teacher, both models share the same input data -- and
this data is the only medium by which the teacher's knowledge can be
demonstrated. Due to the difference in model capacities, the student may not
benefit fully from the same data points on which the teacher is trained. On the
other hand, a human teacher may demonstrate a piece of knowledge with
individualized examples adapted to a particular student, for instance, in terms
of her cultural background and interests. Inspired by this behavior, we design
data augmentation agents with distinct roles to facilitate knowledge
distillation. Our data augmentation agents generate distinct training data for
the teacher and student, respectively. We find empirically that specially
tailored data points enable the teacher's knowledge to be demonstrated more
effectively to the student. We compare our approach with existing KD methods on
training popular neural architectures and demonstrate that role-wise data
augmentation improves the effectiveness of KD over strong prior approaches. The
code for reproducing our results can be found at
https://github.com/bigaidream-projects/role-kd
