Iterative Graph Self-Distillation
  How to discriminatively vectorize graphs is a fundamental challenge that
attracts increasing attentions in recent years. Inspired by the recent success
of unsupervised contrastive learning, we aim to learn graph-level
representation in an unsupervised manner. Specifically, we propose a novel
unsupervised graph learning paradigm called Iterative Graph Self-Distillation
(IGSD) which iteratively performs the teacher-student distillation with graph
augmentations. Different from conventional knowledge distillation, IGSD
constructs the teacher with an exponential moving average of the student model
and distills the knowledge of itself. The intuition behind IGSD is to predict
the teacher network representation of the graph pairs under different augmented
views. As a natural extension, we also apply IGSD to semi-supervised scenarios
by jointly regularizing the network with both supervised and unsupervised
contrastive loss. Finally, we show that finetuning the IGSD-trained models with
self-training can further improve the graph representation power. Empirically,
we achieve significant and consistent performance gain on various graph
datasets in both unsupervised and semi-supervised settings, which well
validates the superiority of IGSD.
