Out-of-Vocabulary Entities in Link Prediction
  Knowledge graph embedding techniques are key to making knowledge graphs
amenable to the plethora of machine learning approaches based on vector
representations. Link prediction is often used as a proxy to evaluate the
quality of these embeddings. Given that the creation of benchmarks for link
prediction is a time-consuming endeavor, most work on the subject matter uses
only a few benchmarks. As benchmarks are crucial for the fair comparison of
algorithms, ensuring their quality is tantamount to providing a solid ground
for developing better solutions to link prediction and ipso facto embedding
knowledge graphs. First studies of benchmarks pointed to limitations pertaining
to information leaking from the development to the test fragments of some
benchmark datasets. We spotted a further common limitation of three of the
benchmarks commonly used for evaluating link prediction approaches:
out-of-vocabulary entities in the test and validation sets. We provide an
implementation of an approach for spotting and removing such entities and
provide corrected versions of the datasets WN18RR, FB15K-237, and YAGO3-10. Our
experiments on the corrected versions of WN18RR, FB15K-237, and YAGO3-10
suggest that the measured performance of state-of-the-art approaches is altered
significantly with p-values <1%, <1.4%, and <1%, respectively. Overall,
state-of-the-art approaches gain on average absolute $3.29 \pm 0.24\%$ in all
metrics on WN18RR. This means that some of the conclusions achieved in previous
works might need to be revisited. We provide an open-source implementation of
our experiments and corrected datasets at at
https://github.com/dice-group/OOV-In-Link-Prediction.
