MulDE: Multi-teacher Knowledge Distillation for Low-dimensional
  Knowledge Graph Embeddings
  Link prediction based on knowledge graph embeddings (KGE) aims to predict new
triples to automatically construct knowledge graphs (KGs). However, recent KGE
models achieve performance improvements by excessively increasing the embedding
dimensions, which may cause enormous training costs and require more storage
space. In this paper, instead of training high-dimensional models, we propose
MulDE, a novel knowledge distillation framework, which includes multiple
low-dimensional hyperbolic KGE models as teachers and two student components,
namely Junior and Senior. Under a novel iterative distillation strategy, the
Junior component, a low-dimensional KGE model, asks teachers actively based on
its preliminary prediction results, and the Senior component integrates
teachers' knowledge adaptively to train the Junior component based on two
mechanisms: relation-specific scaling and contrast attention. The experimental
results show that MulDE can effectively improve the performance and training
speed of low-dimensional KGE models. The distilled 32-dimensional model is
competitive compared to the state-of-the-art high-dimensional methods on
several widely-used datasets.
