Alignahead: Online Cross-Layer Knowledge Extraction on Graph Neural
  Networks
  Existing knowledge distillation methods on graph neural networks (GNNs) are
almost offline, where the student model extracts knowledge from a powerful
teacher model to improve its performance. However, a pre-trained teacher model
is not always accessible due to training cost, privacy, etc. In this paper, we
propose a novel online knowledge distillation framework to resolve this
problem. Specifically, each student GNN model learns the extracted local
structure from another simultaneously trained counterpart in an alternating
training procedure. We further develop a cross-layer distillation strategy by
aligning ahead one student layer with the layer in different depth of another
student model, which theoretically makes the structure information spread over
all layers. Experimental results on five datasets including PPI,
Coauthor-CS/Physics and Amazon-Computer/Photo demonstrate that the student
performance is consistently boosted in our collaborative training framework
without the supervision of a pre-trained teacher model. In addition, we also
find that our alignahead technique can accelerate the model convergence speed
and its effectiveness can be generally improved by increasing the student
numbers in training. Code is available:
https://github.com/GuoJY-eatsTG/Alignahead
