Play the Shannon Game With Language Models: A Human-Free Approach to
  Summary Evaluation
  The goal of a summary is to concisely state the most important information in
a document. With this principle in mind, we introduce new reference-free
summary evaluation metrics that use a pretrained language model to estimate the
information content shared between a document and its summary. These metrics
are a modern take on the Shannon Game, a method for summary quality scoring
proposed decades ago, where we replace human annotators with language models.
We also view these metrics as an extension of BLANC, a recently proposed
approach to summary quality measurement based on the performance of a language
model with and without the help of a summary. Using transformer based language
models, we empirically verify that our metrics achieve state-of-the-art
correlation with human judgement of the summary quality dimensions of both
coherence and relevance, as well as competitive correlation with human
judgement of consistency and fluency.
