Graph-Free Knowledge Distillation for Graph Neural Networks
  Knowledge distillation (KD) transfers knowledge from a teacher network to a
student by enforcing the student to mimic the outputs of the pretrained teacher
on training data. However, data samples are not always accessible in many cases
due to large data sizes, privacy, or confidentiality. Many efforts have been
made on addressing this problem for convolutional neural networks (CNNs) whose
inputs lie in a grid domain within a continuous space such as images and
videos, but largely overlook graph neural networks (GNNs) that handle non-grid
data with different topology structures within a discrete space. The inherent
differences between their inputs make these CNN-based approaches not applicable
to GNNs. In this paper, we propose to our best knowledge the first dedicated
approach to distilling knowledge from a GNN without graph data. The proposed
graph-free KD (GFKD) learns graph topology structures for knowledge transfer by
modeling them with multivariate Bernoulli distribution. We then introduce a
gradient estimator to optimize this framework. Essentially, the gradients
w.r.t. graph structures are obtained by only using GNN forward-propagation
without back-propagation, which means that GFKD is compatible with modern GNN
libraries such as DGL and Geometric. Moreover, we provide the strategies for
handling different types of prior knowledge in the graph data or the GNNs.
Extensive experiments demonstrate that GFKD achieves the state-of-the-art
performance for distilling knowledge from GNNs without training data.
