Multi-view Contrastive Learning for Online Knowledge Distillation
  Previous Online Knowledge Distillation (OKD) often carries out mutually
exchanging probability distributions, but neglects the useful representational
knowledge. We therefore propose Multi-view Contrastive Learning (MCL) for OKD
to implicitly capture correlations of feature embeddings encoded by multiple
peer networks, which provide various views for understanding the input data
instances. Benefiting from MCL, we can learn a more discriminative
representation space for classification than previous OKD methods. Experimental
results on image classification demonstrate that our MCL-OKD outperforms other
state-of-the-art OKD methods by large margins without sacrificing additional
inference cost. Codes are available at https://github.com/winycg/MCL-OKD.
