Does entity abstraction help generative Transformers reason?
  Pre-trained language models (LMs) often struggle to reason logically or
generalize in a compositional fashion. Recent work suggests that incorporating
external entity knowledge can improve LMs' abilities to reason and generalize.
However, the effect of explicitly providing entity abstraction remains unclear,
especially with recent studies suggesting that pre-trained LMs already encode
some of that knowledge in their parameters. We study the utility of
incorporating entity type abstractions into pre-trained Transformers and test
these methods on four NLP tasks requiring different forms of logical reasoning:
(1) compositional language understanding with text-based relational reasoning
(CLUTRR), (2) abductive reasoning (ProofWriter), (3) multi-hop question
answering (HotpotQA), and (4) conversational question answering (CoQA). We
propose and empirically explore three ways to add such abstraction: (i) as
additional input embeddings, (ii) as a separate sequence to encode, and (iii)
as an auxiliary prediction task for the model. Overall, our analysis
demonstrates that models with abstract entity knowledge performs better than
without it. However, our experiments also show that the benefits strongly
depend on the technique used and the task at hand. The best abstraction aware
models achieved an overall accuracy of 88.8% and 91.8% compared to the baseline
model achieving 62.3% and 89.8% on CLUTRR and ProofWriter respectively. In
addition, abstraction-aware models showed improved compositional generalization
in both interpolation and extrapolation settings. However, for HotpotQA and
CoQA, we find that F1 scores improve by only 0.5% on average. Our results
suggest that the benefit of explicit abstraction is significant in formally
defined logical reasoning settings requiring many reasoning hops, but point to
the notion that it is less beneficial for NLP tasks having less formal logical
structure.
