Analysis of GraphSum's Attention Weights to Improve the Explainability
  of Multi-Document Summarization
  Modern multi-document summarization (MDS) methods are based on transformer
architectures. They generate state of the art summaries, but lack
explainability. We focus on graph-based transformer models for MDS as they
gained recent popularity. We aim to improve the explainability of the
graph-based MDS by analyzing their attention weights. In a graph-based MDS such
as GraphSum, vertices represent the textual units, while the edges form some
similarity graph over the units. We compare GraphSum's performance utilizing
different textual units, i. e., sentences versus paragraphs, on two news
benchmark datasets, namely WikiSum and MultiNews. Our experiments show that
paragraph-level representations provide the best summarization performance.
Thus, we subsequently focus oAnalysisn analyzing the paragraph-level attention
weights of GraphSum's multi-heads and decoding layers in order to improve the
explainability of a transformer-based MDS model. As a reference metric, we
calculate the ROUGE scores between the input paragraphs and each sentence in
the generated summary, which indicate source origin information via text
similarity. We observe a high correlation between the attention weights and
this reference metric, especially on the the later decoding layers of the
transformer architecture. Finally, we investigate if the generated summaries
follow a pattern of positional bias by extracting which paragraph provided the
most information for each generated summary. Our results show that there is a
high correlation between the position in the summary and the source origin.
