Contrastive Representation Distillation
  Often we wish to transfer representational knowledge from one neural network
to another. Examples include distilling a large network into a smaller one,
transferring knowledge from one sensory modality to a second, or ensembling a
collection of models into a single estimator. Knowledge distillation, the
standard approach to these problems, minimizes the KL divergence between the
probabilistic outputs of a teacher and student network. We demonstrate that
this objective ignores important structural knowledge of the teacher network.
This motivates an alternative objective by which we train a student to capture
significantly more information in the teacher's representation of the data. We
formulate this objective as contrastive learning. Experiments demonstrate that
our resulting new objective outperforms knowledge distillation and other
cutting-edge distillers on a variety of knowledge transfer tasks, including
single model compression, ensemble distillation, and cross-modal transfer. Our
method sets a new state-of-the-art in many transfer tasks, and sometimes even
outperforms the teacher network when combined with knowledge distillation.
Code: http://github.com/HobbitLong/RepDistiller.
