On Self-Distilling Graph Neural Network
  Recently, the teacher-student knowledge distillation framework has
demonstrated its potential in training Graph Neural Networks (GNNs). However,
due to the difficulty of training over-parameterized GNN models, one may not
easily obtain a satisfactory teacher model for distillation. Furthermore, the
inefficient training process of teacher-student knowledge distillation also
impedes its applications in GNN models. In this paper, we propose the first
teacher-free knowledge distillation method for GNNs, termed GNN
Self-Distillation (GNN-SD), that serves as a drop-in replacement of the
standard training process. The method is built upon the proposed neighborhood
discrepancy rate (NDR), which quantifies the non-smoothness of the embedded
graph in an efficient way. Based on this metric, we propose the adaptive
discrepancy retaining (ADR) regularizer to empower the transferability of
knowledge that maintains high neighborhood discrepancy across GNN layers. We
also summarize a generic GNN-SD framework that could be exploited to induce
other distillation strategies. Experiments further prove the effectiveness and
generalization of our approach, as it brings: 1) state-of-the-art GNN
distillation performance with less training cost, 2) consistent and
considerable performance enhancement for various popular backbones.
