Exploiting Structured Knowledge in Text via Graph-Guided Representation
  Learning
  In this work, we aim at equipping pre-trained language models with structured
knowledge. We present two self-supervised tasks learning over raw text with the
guidance from knowledge graphs. Building upon entity-level masked language
models, our first contribution is an entity masking scheme that exploits
relational knowledge underlying the text. This is fulfilled by using a linked
knowledge graph to select informative entities and then masking their mentions.
In addition we use knowledge graphs to obtain distractors for the masked
entities, and propose a novel distractor-suppressed ranking objective which is
optimized jointly with masked language model. In contrast to existing
paradigms, our approach uses knowledge graphs implicitly, only during
pre-training, to inject language models with structured knowledge via learning
from raw text. It is more efficient than retrieval-based methods that perform
entity linking and integration during finetuning and inference, and generalizes
more effectively than the methods that directly learn from concatenated graph
triples. Experiments show that our proposed model achieves improved performance
on five benchmark datasets, including question answering and knowledge base
completion tasks.
