Relational World Knowledge Representation in Contextual Language Models:
  A Review
  Relational knowledge bases (KBs) are commonly used to represent world
knowledge in machines. However, while advantageous for their high degree of
precision and interpretability, KBs are usually organized according to
manually-defined schemas, which limit their expressiveness and require
significant human efforts to engineer and maintain. In this review, we take a
natural language processing perspective to these limitations, examining how
they may be addressed in part by training deep contextual language models (LMs)
to internalize and express relational knowledge in more flexible forms. We
propose to organize knowledge representation strategies in LMs by the level of
KB supervision provided, from no KB supervision at all to entity- and
relation-level supervision. Our contributions are threefold: (1) We provide a
high-level, extensible taxonomy for knowledge representation in LMs; (2) Within
our taxonomy, we highlight notable models, evaluation tasks, and findings, in
order to provide an up-to-date review of current knowledge representation
capabilities in LMs; and (3) We suggest future research directions that build
upon the complementary aspects of LMs and KBs as knowledge representations.
