Enriching Transformers with Structured Tensor-Product Representations
  for Abstractive Summarization
  Abstractive summarization, the task of generating a concise summary of input
documents, requires: (1) reasoning over the source document to determine the
salient pieces of information scattered across the long document, and (2)
composing a cohesive text by reconstructing these salient facts into a shorter
summary that faithfully reflects the complex relations connecting these facts.
In this paper, we adapt TP-TRANSFORMER (Schlag et al., 2019), an architecture
that enriches the original Transformer (Vaswani et al., 2017) with the
explicitly compositional Tensor Product Representation (TPR), for the task of
abstractive summarization. The key feature of our model is a structural bias
that we introduce by encoding two separate representations for each token to
represent the syntactic structure (with role vectors) and semantic content
(with filler vectors) separately. The model then binds the role and filler
vectors into the TPR as the layer output. We argue that the structured
intermediate representations enable the model to take better control of the
contents (salient facts) and structures (the syntax that connects the facts)
when generating the summary. Empirically, we show that our TP-TRANSFORMER
outperforms the Transformer and the original TP-TRANSFORMER significantly on
several abstractive summarization datasets based on both automatic and human
evaluations. On several syntactic and semantic probing tasks, we demonstrate
the emergent structural information in the role vectors and improved syntactic
interpretability in the TPR layer outputs. Code and models are available at
https://github.com/jiangycTarheel/TPT-Summ.
