Online Adversarial Distillation for Graph Neural Networks
  Knowledge distillation has recently become a popular technique to improve the
model generalization ability on convolutional neural networks. However, its
effect on graph neural networks is less than satisfactory since the graph
topology and node attributes are likely to change in a dynamic way and in this
case a static teacher model is insufficient in guiding student training. In
this paper, we tackle this challenge by simultaneously training a group of
graph neural networks in an online distillation fashion, where the group
knowledge plays a role as a dynamic virtual teacher and the structure changes
in graph neural networks are effectively captured. To improve the distillation
performance, two types of knowledge are transferred among the students to
enhance each other: local knowledge reflecting information in the graph
topology and node attributes, and global knowledge reflecting the prediction
over classes. We transfer the global knowledge with KL-divergence as the
vanilla knowledge distillation does, while exploiting the complicated structure
of the local knowledge with an efficient adversarial cyclic learning framework.
Extensive experiments verified the effectiveness of our proposed online
adversarial distillation approach.
