Distill2Vec: Dynamic Graph Representation Learning with Knowledge
  Distillation
  Dynamic graph representation learning strategies are based on different
neural architectures to capture the graph evolution over time. However, the
underlying neural architectures require a large amount of parameters to train
and suffer from high online inference latency, that is several model parameters
have to be updated when new data arrive online. In this study we propose
Distill2Vec, a knowledge distillation strategy to train a compact model with a
low number of trainable parameters, so as to reduce the latency of online
inference and maintain the model accuracy high. We design a distillation loss
function based on Kullback-Leibler divergence to transfer the acquired
knowledge from a teacher model trained on offline data, to a small-size student
model for online data. Our experiments with publicly available datasets show
the superiority of our proposed model over several state-of-the-art approaches
with relative gains up to 5% in the link prediction task. In addition, we
demonstrate the effectiveness of our knowledge distillation strategy, in terms
of number of required parameters, where Distill2Vec achieves a compression
ratio up to 7:100 when compared with baseline approaches. For reproduction
purposes, our implementation is publicly available at
https://stefanosantaris.github.io/Distill2Vec.
