Abstractive Text Summarization based on Language Model Conditioning and
  Locality Modeling
  We explore to what extent knowledge about the pre-trained language model that
is used is beneficial for the task of abstractive summarization. To this end,
we experiment with conditioning the encoder and decoder of a Transformer-based
neural model on the BERT language model. In addition, we propose a new method
of BERT-windowing, which allows chunk-wise processing of texts longer than the
BERT window size. We also explore how locality modelling, i.e., the explicit
restriction of calculations to the local context, can affect the summarization
ability of the Transformer. This is done by introducing 2-dimensional
convolutional self-attention into the first layers of the encoder. The results
of our models are compared to a baseline and the state-of-the-art models on the
CNN/Daily Mail dataset. We additionally train our model on the SwissText
dataset to demonstrate usability on German. Both models outperform the baseline
in ROUGE scores on two datasets and show its superiority in a manual
qualitative analysis.
