E.T.: Entity-Transformers. Coreference augmented Neural Language Model
  for richer mention representations via Entity-Transformer blocks
  In the last decade, the field of Neural Language Modelling has witnessed
enormous changes, with the development of novel models through the use of
Transformer architectures. However, even these models struggle to model long
sequences due to memory constraints and increasing computational complexity.
Coreference annotations over the training data can provide context far beyond
the modelling limitations of such language models. In this paper we present an
extension over the Transformer-block architecture used in neural language
models, specifically in GPT2, in order to incorporate entity annotations during
training. Our model, GPT2E, extends the Transformer layers architecture of GPT2
to Entity-Transformers, an architecture designed to handle coreference
information when present. To that end, we achieve richer representations for
entity mentions, with insignificant training cost. We show the comparative
model performance between GPT2 and GPT2E in terms of Perplexity on the CoNLL
2012 and LAMBADA datasets as well as the key differences in the entity
representations and their effects in downstream tasks such as Named Entity
Recognition. Furthermore, our approach can be adopted by the majority of
Transformer-based language models.
