Extract the Knowledge of Graph Neural Networks and Go Beyond it: An
  Effective Knowledge Distillation Framework
  Semi-supervised learning on graphs is an important problem in the machine
learning area. In recent years, state-of-the-art classification methods based
on graph neural networks (GNNs) have shown their superiority over traditional
ones such as label propagation. However, the sophisticated architectures of
these neural models will lead to a complex prediction mechanism, which could
not make full use of valuable prior knowledge lying in the data, e.g.,
structurally correlated nodes tend to have the same class. In this paper, we
propose a framework based on knowledge distillation to address the above
issues. Our framework extracts the knowledge of an arbitrary learned GNN model
(teacher model), and injects it into a well-designed student model. The student
model is built with two simple prediction mechanisms, i.e., label propagation
and feature transformation, which naturally preserves structure-based and
feature-based prior knowledge, respectively. In specific, we design the student
model as a trainable combination of parameterized label propagation and feature
transformation modules. As a result, the learned student can benefit from both
prior knowledge and the knowledge in GNN teachers for more effective
predictions. Moreover, the learned student model has a more interpretable
prediction process than GNNs. We conduct experiments on five public benchmark
datasets and employ seven GNN models including GCN, GAT, APPNP, SAGE, SGC,
GCNII and GLP as the teacher models. Experimental results show that the learned
student model can consistently outperform its corresponding teacher model by
1.4% - 4.7% on average. Code and data are available at
https://github.com/BUPT-GAMMA/CPF
