{'fraction_non_alphanumeric': 0.03588717118805275, 'fraction_numerical': 0.015870987890918897, 'mean_word_length': 4.817971589032045, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 5, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
News Summarization and Evaluation in the Era of GPT-3

The recent success of zero- and few-shot prompting with models like GPT-3 has led to a paradigm shift in NLP research. In this paper, we study its impact on text summarization, focusing on the classic benchmark domain of news summarization. First, we investigate how zero-shot GPT-3 compares against fine-tuned models trained on large summarization datasets. We show that not only do humans overwhelmingly prefer GPT-3 summaries, but these also do not suffer from common dataset-specific issues such as poor factuality. Next, we study what this means for evaluation, particularly the role of gold standard test sets. Our experiments show that both reference-based and reference-free automatic metrics, e.g. recently proposed QA- or entailment-based factuality approaches, cannot reliably evaluate zero-shot summaries. Finally, we discuss future research challenges beyond generic summarization, specifically, keyword- and aspect-based summarization, showing how dominant fine-tuning approaches compare to zero-shot prompting. To support further research, we release: (a) a corpus of 10K generated summaries from fine-tuned and zero-shot models across 4 standard summarization benchmarks, (b) 1K human preference judgments and rationales comparing different systems for generic- and keyword-based summarization.


Introduction
Fine-tuning pre-trained models on domain-specific datasets has been the leading paradigm in text summarization research in recent years (Lewis et al., 2020;Zhang et al., 2020;Raffel et al., 2020). These models generate high-quality summaries on standard benchmarks, but still require sizeable training datasets to adapt to new settings, e.g., sum- Figure 1: Examples of zero-shot summarization using GPT-3. We can generate summaries following style constraints or queries included in the prompts, allowing us to emulate a range of existing fine-tuned systems. marizing data from a new source domain or producing a summary in a different style. The success of prompt-based models (GPT-3 (Brown et al., 2020), T0 (Sanh et al., 2022), PaLM (Chowdhery et al., 2022), etc.) provides an alternative approach, namely learning from natural language task instructions and/or a few demonstrative examples in the context without updating model parameters. While recent work (Zhao et al., 2021;Min et al., 2022;Ye and Durrett, 2022) has analyzed and evaluated this paradigm across a number of tasks, it has only been studied for text summarization with unreliable automatic metrics (He et al., 2022;Chowdhery et al., 2022;Ouyang et al., 2022) or in non-standard settings (Saunders et al., 2022).
In this paper, we conduct the first systematic study of the impact of prompt-based models on the text summarization research space, using an Instruct-tuned 175B GPT-3 model (text-davinci-002) (Brown et al., 2020;Ouyang et al., 2022) as a case study. Figure 1 shows that GPT-3 summaries are extremely high-quality and adaptable to different summarization settings. Starting from these observations, we aim to answer three main questions. First, how do prompt-based GPT-3 summaries compare to those obtained from state-ofthe-art fine-tuned summarization models (Zhang et al., 2020;Liu et al., 2022)? We compare these approaches using A/B testing on a new corpus of recent news articles, and find that our study participants overwhelmingly prefer zero-shot GPT-3 summaries across two different "styles" with different prompts (three-sentence and single-sentence). Moreover, these zero-shot summaries do not suffer from limitations due to low-quality training data that plague fine-tuned generic summarization models (Maynez et al., 2020;Goyal et al., 2022).
Second, are existing automatic metrics wellsuited to evaluating zero-shot summaries? Recent work has shown that classic reference-based such as ROUGE (Lin, 2004) and BERTScore (Zhang* et al., 2020) are unreliable when small improvements are reported (Peyrard, 2019;Fabbri et al., 2021); however large differences, on the order of say 5 ROUGE points or greater, are considered to be correlated with human preferences (Bhandari et al., 2020;Deutsch et al., 2022). However, we find that the same is no longer true when evaluating zero-shot GPT-3 summaries. These summaries score much lower on automatic metrics (7 ROUGE-L points on average) than all prior state-of-the-art models while comfortably outperforming them on human evaluation. Furthermore, we show that recent reference-free metrics, e.g. QA-based metrics (Fabbri et al., 2022;Durmus et al., 2020) and trained factuality models (Kryscinski et al., 2020;Goyal and Durrett, 2020), similarly fail to adapt to this shift from the fine-tuned to zero-shot summarization space and need to be re-visited.
Finally, how can zero-shot summaries be used for use cases beyond generic summarization? We focus on keyword-based and aspect-based summarization. For keyword-based summarization, we find that zero-shot GPT-3 consistently generates more coherent and keyword-relevant summaries compared to current fine-tuned alternatives: crowd annotators prefer GPT-3 summaries over a baseline model (He et al., 2020) 70% of the time. We observe mixed results for the aspect-based setting, where GPT-3 summaries show frequent failure cases when prompted with simple prompts for aspect-based summarization.
Taken together, this evidence suggests that GPT-3 represents a fundamental paradigm shift in sum-marization, changing the what data we need (or don't need) and what approaches we can now explore. Evaluating these systems as they progress further will require a new framework distinct from the automatic metrics that have dominated the last decade of summarization research.

Current Paradigms for Summarization
Recent zero-and few-shot prompting based models (Brown et al., 2020;Sanh et al., 2022), have shown impressive generalization capabilities on unseen tasks specified using prompts alone and without performing any gradient updates (Mishra et al., 2022). In this work, we want to compare their text summarization performance against the current state-of-the-art models.

Pre-trained LM (Task-specific models trained for each dataset)
Fine-tuned on summ. datasets

Instruction-tuned on multiple tasks
Zero-shot prompting Zero-shot prompting   Figure 2 shows the broad categories of all available summarization approaches, including current SOTA models and prompting-based models. The former set consists of fine-tuned language models, trained on a large number of article-summary pairs (e.g. BART (Lewis et al., 2020), PEGASUS (Zhang et al., 2020), BRIO (Liu et al., 2022)) to obtain dataset-specific systems. This category also includes models aimed at tasks beyond generic summarization, such as keyword-or query-based summarization, that still rely on standard datasets for training (He et al., 2020).
On the other extreme are zero-or few-shot models, (e.g. GPT3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022)), that are not explicitly trained for any particular task, as discussed above. Recent work (Ouyang et al., 2022;Wei et al., 2022;Sanh et al., 2022) has improved on these models by introducing instruction-tuned models. Here, pre-trained language models are fine-tuned on multiple tasks (which may include summarization) using instruction templates in order to align their  Table 1: Basic statistics of standard summarization datasets: CNN/DM (Hermann et al., 2015;Nallapati et al., 2016), XSum (Narayan et al., 2018), Newsroom (Grusky et al., 2018). These show large variance in their summary properties and fundamentally differ in their definition of the "gold" standard.
training with inference time usage.
In this work, we compare the summarization performance of three models that are representative of this space of options: 1. OpenAI's text-davinci-002, a GPT-3 model (Brown et al., 2020) from the Instruct series (Ouyang et al., 2022). While we do not know the exact training details for this release of the model, the previous in the series (textdavinci-001) was fine-tuned on a combination of prompts submitted to their API and labeler prompts spanning multiple tasks. These tasks include summarization but not (to our knowledge) standard summarization datasets like CNN/DM (Hermann et al., 2015;Nallapati et al., 2016) or XSum (Narayan et al., 2018).
We choose the text-davinci-002 version for our experiments in order to benchmark the best available prompt-based model. 2 We refer to this approach as GPT3-D2. (Liu et al., 2022), a fine-tuned summarization model that reports state-of-the art results on both CNN/DM and XSum. We will use versions of this model fine-tuned on each of these two datasets.

Using GPT3-D2 for zero-shot summarization
Fine-tuned models largely follow the "style" of reference summaries in their training data, and hence, The three African nations on the UN Security Council condemned reports of discrimination against African citizens at the Ukrainian border during a meeting at the UN HQ in New York City Monday.
The United Nations Security Council condemned the reports of discrimination against African citizens at the Ukrainian border. The African Union has said it is "disturbed" by the reports of segregation against Africans in Ukraine, which it described as "shockingly racist." The article discusses the reports of discrimination against African citizens at the Ukrainian border. The representatives from the three African nations on the UN Security Council condemned the reports and called for the mistreatment of African peoples on Europe's borders to cease immediately. Foreign students attempting to flee Ukraine after Russia invaded the country told CNN that they experienced racial discrimination at the Ukrainian border.
Prompt: Summarize the article in N sentences.
Article: https://www.cnn.com/2022/03/01/africa/africa-condemnsracism-ukraine-intl/index.html Figure 3: Illustration of length control using the task description / prompt for GPT3-D2. We found that the generated summaries followed the given sentence length constraint 98% of the time, allowing us to generate different length summaries emulating different datasets. generated summaries show large variance between datasets (see Table 1 for basic summary statistics of standard summarization datasets). To ensure fair comparison between these and zero-shot GPT3-D2, we adapt the latter's prompt to align with datasetspecific styles. Specifically, we follow prior work (Sanh et al., 2022) and use sentence-count length prompts to adapt to each dataset. Although these datasets also differ along other attributes, e.g. CNN/DM is leadbiased whereas XSum requires inferences drawing from a whole article, we do not attempt to control any other attributed of the summary. Figure 3 shows an example of different length GPT3-D2 summaries for the same news article, using the following prompt format: 3 Article: {{article}} Summarize the above article in N sentences.
We found that GPT3-D2 summaries faithfully follow the given length constraint in 98% of the test instances used in our human study data in Section 3.
Given this setup, we first compare the summary quality of the three summarization models through a human annotation study (Section 3). Then, we evaluate the current suite of summarization metrics for zero-shot summarization (Section 4). Finally, in Section 5, we briefly discuss GPT3-D2 performance on summarization tasks beyond generic summarization and new challenges.

Human evaluation of GPT3-D2 summaries
Generated summaries of fine-tuned models (Lewis et al., 2020;Zhang et al., 2020;Liu et al., 2022) emulate gold-standard summaries in their training datasets. In contrast, prompt-based GPT3-D2 models generate summaries based on how the given task description surfaces behavior learned during pre-training or instruction-tuning. In this section, we ask: how do these paradigms compare? Does learning from gold summaries lead to a better summarization model? To answer this, we conduct a human study to compare outputs of our 3 representative models and collect human preferences of quality.

Experimental Setup
Datasets for fine-tuning We choose two standard fine-tuning datasets whose summaries differ along multiple dimensions such as length and abstractiveness: 1. CNN/DM (Hermann et al., 2015;Nallapati et al., 2016) contains reference summaries that are approximately 3-4 sentences long. Summaries in this dataset are highly extractive and lead-biased.
2. XSum (Narayan et al., 2018) contains 1 sentence summaries of BBC news articles. In this dataset, references summaries, and consequently generated summaries from fine-tuned models are highly abstractive.
Datasets for evaluation Because GPT3-D2's pretraining and instruction-tuning datasets are unknown, it may have been trained on existing articles and summaries in the test splits of these standard benchmarks. We therefore run our human study on 100 recent articles from CNN 4 and BBC, collected between March 1, 2022 and June 31, 2022. We call these CNN-2022 andBBC-2022 respectively. 4 Although the BRIO's CNN/DM model also includes Daily-Mail data in its training, we do not use this news source in our study as it is now widely considered to be unreliable. E.g. according to Media Bias / Fact Check site, DM's factual reporting is rated 'low' https://mediabiasfactcheck.com/ daily-mail/.
Model details We use the publicly released BRIO-XSum and BRIO-CNN/DM models to generate summaries. 5 For T0, we use a prompt we selected from its prompt repository for CNN/DM and XSum datasets. 6 Finally, to generate GPT3-D2 summaries, we set N = 3 for CNN and N = 1 for BBC in our standard sentence-count prompt template from Section 2.
For a maximally fair comparison in this "realistic" setting, we take some additional steps to improve the output of BRIO-XSum. In order to automate dataset creation, XSum removes the first sentence from news articles to use as the gold summary for training, then treats the rest of the sentences as the article to summarize. This setup differs from the real world usage of summarization systems where the complete article is summarized. Due to this mismatch, BRIO-XSum often generates very low quality outputs, e.g. All images: Strule Shared Education Campus in Figure 4, for around 30% of the articles. We manually identify these examples and first attempt to fix them by selecting a summary without such obvious failures from further down the beam (we use beam size = 10). However, if we cannot find a "better" summary, we remove the first sentence of the article and re-sample a new summary to align with its noisy training. This latter strategy often results in factually incorrect summary generations, as is well documented in prior research (Maynez et al., 2020;Goyal and Durrett, 2021).
Design of the human study We design an A/B test to collect preference annotations. For each given article, annotators are shown summaries from all three summarization systems (BRIO, T0 and GPT3-D2). They are then asked to select their most and least preferred summary or summaries. In addition to these multiple choice questions, we also ask for a free-text justification of both choices.
We make two design decisions for our human study: first, we do not provide annotators with specific definitions of summary quality to avoid introducing our own biases. It is also quite challenging to produce a unified definition of quality for the very different "styles" of summaries evaluated in this study. Instead, we ask them to rely on their own preferences based on summaries they would like to see if they were browsing the web, which Mortgage rates have fallen slightly in the past few weeks, but they are still well above last year's levels. This is making it difficult for many prospective buyers to afford a home. The Fed has been raising interest rates in an effort to control inflation, and more rate hikes are expected. The 30-year, fixed-rate mortgage averaged 5.09% in the week ending June 2. It's the third consecutive week of declines. But rates are still above 5% and home prices are well above where they were this time last year. Rising prices have pushed some prospective buyers to the sidelines.
30-year, fixed-rate mortgage averaged 5.09% in the week ending June 2. It is still well above the 2.99% average from this time last year. Rising prices had already been pushing many prospective buyers to the sidelines.  Figure 4: Examples of CNN-style and BBC/XSum-style summaries for the three systems. For CNN, we observe that models fine-tuned on the CNN/DM training set reflect its dataset biases; summaries are highly extractive, specific and lead-biased. On the other hand, GPT3-D2 summaries contain fewer specific details but cover more content.
we believe to be a representative scenario for nonexpert consumers of news summaries. Detailed task instructions are included in Appendix F.
Second, we allow multiple selections for both the best and worst summary questions to cater to scenarios in which different summarization systems output similar quality summaries without meaningful differences.
We hire crowd annotators through Prolific. For both CNN and BBC, we recruit 60 unique participants to annotate the 100 summaries in each dataset. Each annotator was asked to annotate 5 articles and each article was annotated by 3 annotators. Additionally, we use the Prolific's demographic filters to restrict participation to USA (or UK) residents for CNN (or BBC). We anticipate that residents from these respective countries are better positioned to understand country-specific news events and evaluate their summaries. Participants were paid approximately $11/hr for their work.

Results
Differences between summarization systems Figure 4 shows examples of generated summaries from all three summarization systems for both CNN and BBC articles. For CNN, we observe that fine-tuned BRIO summaries tend to be highly extractive and generally include a high number of named entities (dates, percentages, names), reflecting the data it was trained on. In contrast, GPT3-D2 sum-

Model
Length Statistics % novel n-gms #NEs per #sent #words/sent n = 1 n = 2 100 words  Table 2: Statistics for generated summaries evaluated in the human study across all datasets and summarization systems. We observe that GPT3-D2 generated summaries nearly always follow the sentence length constraints in their prompts.
maries are more abstractive and less specific, but provide a more exhaustive overview of the article content. Table 2 provides quantitative evidence of this; we use percentage of novel n-grams to measure abstractiveness, and number of named entities per 100 words to measure specificity. For BBC, we observe inverse trends where BRIO and T0 are more abstractive compared to GPT3-D2. Again, this can be attributed to the XSum training data used to train both these prior models. For GPT3-D2 summaries, on the other hand, the level of abstractiveness does not differ between datasets. Finally, Table 2 shows that GPT3-D2 summaries tend to have longer sentences, and therefore  Which systems do humans prefer? Results of our human study are summarized in Table 3. We report the percentage of times a particular system is the most/least preferred model according to majority vote combining all three annotator's choices. 7 Across both datasets and styles, we observe a clear preference for GPT3-D2 summaries compared to the other two models. In fact, in both scenarios, the GPT3-D2 outperforms the next best model by at least 20 percentage points. This improvement is statistically significant according to a paired bootstrap test (CNN p−value = 2 × 10 −3 , BBC p−value = 6 × 10 −4 ). Note that the next best model differs between the two datasets. For BBC, annotators prefer T0 summaries over BRIO. Annotator rationales often mentioned misleading or incorrect information as the primarily reason for selecting BRIO as the worst summary, confirming the issues that have been observed with XSum-trained models (Maynez et al., 2020;Pagnoni et al., 2021;Goyal and Durrett, 2021). Although T0 also includes XSum training data, we hypothesize that its multi-task framework helps offset the noisy signal from XSum.
In contrast, annotators rate T0 as the worst summarization system for CNN. The most common rationales for these were shorter length and inclusion of irrelevant details, e.g. long quotes, while missing key points. Some annotators also commented that these T0 summaries were less coherent compared to the other models. Interestingly, we did not observe similar complaints for the singlesentence T0 summaries for BBC. : Annotator vote distribution for best and worst summaries across all datasets and models. Although GPT3-D2 is the clear winner according to majority vote, this choice is unanimous for less than 30% of the articles. This demonstrates the inherent variance in different annotators' definitions of "best summary", especially when comparing high-quality summaries from strong models.
Do annotators agree with each other? To study this, we plot the distribution of annotator votes for each summarization system and dataset in Figure 5. Additionally, we report the inter-annotator agreement, measured using Krippendorff's alpha with MASI distance (Passonneau, 2006), to account for multiple selections of best or worst summary allowed in our study design. The vote distribution shows that although more annotators prefer GPT3-D2 summaries, this choice is only unanimous, i.e. supported by all three annotators, for less that 30% of the annotated articles. Conversely, although BRIO (or T0) summaries are less preferred than GPT3-D2 for the CNN (or BBC) dataset on aggregate, they were voted as the best summary by at least one annotator for more than 60% of the articles. This demonstrate two things: first, when comparing summaries from two strong models, the choice is inherently ambiguous (similar observations in Clark et al. (2021)). Second, these results and the diversity in the written rationales, show that there does not exist a universal definition of a "good" summary and that different summary properties appeal to different annotators. Regardless, the aggregate preference for GPT3-D2 is high enough across the board to give us confidence in its strength.
How do these results impact the field? Progress in text summarization research in the last five years has been enabled by the construction of large-scale text summarization  Table 4: Performance of different summarization systems measured using reference-based automatic metrics. Across all datasets, we observe that automatic metrics report substantially worse results for GPT3-D2 summaries compared to fine-tuned models. This directly contradicts the human preference results from Section 3, demonstrating that these reference-based metrics cannot reliably compare the quality of zero-shot summaries against fine-tuned summaries.
datasets that involved scraping news articles and pairing them with any available summary-like data (Hermann et al., 2015;Narayan et al., 2018;Grusky et al., 2018). The CNN/DM dataset considers bullet points accompanying news articles as its summary. These "gold" standard summaries provided useful training signal to train impressive supervised models (Lewis et al., 2020;Zhang et al., 2020;Liu et al., 2022) and hence, their quality or alignment with human preferences was largely ignored.
We found that, despite its popularity, XSum is largely unsuitable for fine-tuning models like BRIO for realistic summarization settings. Even though a CNN/DM-trained BRIO model performed better, the results of our human study question the continued utility of hill-climbing on this dataset, as it seems users may simply prefer a different style of summary altogether. In fact, this preference for GPT3-D2 is much larger than incremental improvements reported in other human evaluation settings, e.g. improvements on XSum on the GENIE leaderboard (Khashabi et al., 2021). Furthermore, as we we will see in Section 5, the greater flexibility of GPT3-D2 compared to these systems makes it more suitable for news summarization tasks beyond generic summarization.
If a system designer collects a large-scale dataset of high-quality summaries that they wish to emu-late, we believe a fine-tuned system may outperform GPT3-D2. However, better-trained models on datasets collected via "incidental" supervision are less likely to help.

Can current automatic metrics evaluate GPT3-D2 summaries?
Automatic metrics proposed for summarization evaluation can be broadly divided into two categories: (1) reference-based, that compare generated summaries against available gold summaries, and (2) reference-free that only rely on the input document. Here, we compare their performance at evaluating zero-shot GPT3-D2 summaries.
Experimental Setup We evaluate automatic metrics using summaries from 4 different summarization datasets, listed in Table 1. For each dataset, we construct our evaluation sets by randomly sampling 500 8 articles from the standard test split. 9 We compare the same 3 summarization systems from Section 3 in our analysis. Additionally, we also  Table 5: Performance of different summarization systems, as scored by automatic reference-free evaluation metrics from the summarization literature. Similar to reference-based metrics, these also generally fail to produce the same system rankings as human preferences reliably across datasets.
report results using the fine-tuned PEGASUS model (Zhang et al., 2020), as BRIO fine-tuned models are not available for all datasets. We publicly release this corpus of summarization outputs to standardize the test sets and support future research into GPT3-D2 based summarization. Link: https://tagoyal.github.io/ zeroshot-news-annotations.html.

Reference-based metrics
Here, we study if the gold summaries of the standard datasets are useful for evaluation, especially when evaluating zero-shot summaries that are not trained to emulate the gold. We benchmark the performance of 3 different summarization metrics: (1) overlap-based metrics, specifically ROUGE (2) similarity-based metrics, that compute similarity between embeddings representations of generated and reference summaries. Specifically, we report BERTScore (Zhang* et al., 2020) and MoverScore (Zhao et al., 2019). (3) a QA-based metric, specifically QAEval (Deutsch et al., 2021). Although most QAmetrics are reference-free (discussed in Section 4.2), QAEval uses the reference summaries to indicate saliency. We report both exact match (EM) and F1 components of QAEval.
Results Table 4 outlines the results. It shows that BRIO and PEGASUS models, fine-tuned to emulate the reference summaries, outperform GPT3-D2 summaries according to all referencebased automatic metrics. The difference in their assigned scores is very high, e.g. >7 ROUGE-L points between GPT3-D2 and BRIO. For comparison, these reported scores for GPT3-D2 are even lower than the trivial Lead-3 baseline reported in prior work (Fabbri et al., 2021;Grusky et al., 2018). This clearly demonstrates that current automatic reference-based metrics cannot be used to reliably measure summary quality under the zeroshot prompting paradigm.
Amongst prompting-based models, we observe that T0 summaries report better metric scores than GPT3-D2 for all datasets except Newsroom. Interestingly, out of the four datasets evaluated here, Newsroom is the only one not used to train the T0 model. This further shows that access to datasetspecific reference summaries during training improves performance according to these metrics, rendering them unsuitable for zero-shot evaluation.

Reference-free metrics
Next, we investigate whether current reference-free evaluation metrics reflect the human preference rankings between summarization systems, as observed in Section 3. Here, we study 2 categories of metrics: (1) quality metrics, specifically SU-PERT (Gao et al., 2020), which evaluates generated summaries against automatically identified salient sentences in the input, and BLANC (Vasilyev et al., 2020), which evaluates summaries on language un-derstanding tasks. We refer readers to the original papers for detailed explanation of these. (2) factuality metrics, that are evaluate whether generated summaries contain incorrect information with respect to the source article. We report the performance of summarization systems using two QAbased metrics: QuestEval (Scialom et al., 2021) and QAFactEval (Fabbri et al., 2022). Additionally, we also benchmark entailment-based metrics: FactCC (Kryscinski et al., 2020), DAE (Goyal andDurrett, 2020, 2021) and SummaC (Laban et al., 2022). 10 These entailment-based models are designed for classification into factual or non-factual; therefore, we use P (factual | article, summary) to score generated summaries.
Results Table 5 outlines the scores for each summarization system according to the above referencefree metrics. Ideally, we want the relative rankings of different systems according to these metrics to correspond to human preferences, i.e. GPT3-D2 > BRIO > T0 for CNN/DM 11 and GPT3-D2 > T0 > BRIO for XSum. 12 Overall, we observe that none of the referencefree metrics we evaluate follow these trends for both CNN/DM and XSum datasets. In particular, we observe that GPT3-D2 summaries report low factuality scores (except XSum) even though we rarely found any factual errors in our qualitative analysis of its generated summaries.
Interestingly, we noticed a roughly inverse relation to abstractiveness; summarization systems that generated more abstractive summaries (see Table  2) were generally scored lower by all automatic reference-based metrics. For instance, GPT3-D2 is scored lower than BRIO by both quality metrics for all datasets except XSum; the latter is the only dataset for which GPT3-D2 summaries are less abstractive. Such shortcomings of reference-free evaluation metrics due to spurious correlations have also been studied in prior work (Durmus et al., 2022). These issues become more exaggerated when the summarization systems being compared exhibit very different properties. 10 Exact model versions and configurations used for these are outlined in Appendix A. 11 Although the human study in Section 3 is only run on CNN articles, the underlying fine-tuned model is same for both CNN and DM. Therefore, it we can reasonably expect it to display similar quality differences with respect to GPT3-D2.
12 Note that while annotators were not explicitly asked to rate factuality, we instructed them to carefully check factuality and appropriately downvote non-factual summaries.
Discussion On the surface, the failure of reference-free metrics at evaluating GPT3-D2 summaries is more surprising that reference-based metrics as the later explicitly compares generated summaries with references that GPT3-D2 is not trained to imitate. Therefore, GPT3-D2 understandably scores lower than fine-tuned systems.
However, we note two different issues with reference-free metrics: (1) Some of these, e.g. FactCC and DAE, use reference summaries as positive examples to train the metric. Therefore, although "reference-free" at test time, they are still trained to reward the summary properties seen in the standard summarization benchmarks. (2) Even completely reference-free metrics, e.g. QuestEval and QAFactEval, have only been evaluated on reference-based benchmarks and fine-tuned models. Therefore, the choice of different components, such as question answering or question generation models to use, etc. has been dictated by the error space of prior fine-tuned models (Tang et al., 2022). These decisions also now need to be re-visited to incorporate GPT3-D2 evaluation; we leave this for future work.

Beyond Generic Summarization
Previously, we observed that GPT3-D2 models faithfully follow simple "style" instructions in the given prompts. This provides a promising direction to tackle other use cases in news summarization beyond the generic summarization task from Section 3. Different users can have very different information needs from the same article, all of which cannot be satisfied with a single generic summary. Prior work has introduced several task formulations to address this gap, including keyword-focused (He et al., 2020), query-focused (Baumel et al., 2014;He et al., 2020), or aspect-focused summarization (Krishna and Srinivasan, 2018;Ahuja et al., 2022), amongst others. Here, we evaluate GPT3-D2 performance at two of these use cases.
In keyword-based summarization, the output summaries must succinctly summarize the input document focusing on a given keyword; these generally correspond to specific entities or events directly mentioned in the document. In contrast, the control units in aspect-based summarization are high-level topics that can be common across multiple similar types of documents. For e.g., for the input article in Figure 1, Donald Trump or Russian

Article: Republican defenders of Donald Trump won't or can't answer the questions that are at the root of the intrigue over why classified documents were at Mar-a-Lago and the troubling question of whether national security was put at risk. […] At one end of the scale is Missouri Sen. Josh Hawley, a firm Trump backer who is often mentioned as a future presidential candidate once the ex-President finally exits the political stage. Hawley promoted Trump's line that the key issue was the FBI search of an ex-President's home, not what he might have kept there. "I don't know what he has or doesn't have, but what I have a lot of concerns about is the FBI raiding the home of the likely presidential nominee in 2024, the former President of the United States, and particularly as we learn more and more about what's going on with the FBI," Hawley told CNN's Manu Raju on Wednesday. […] Ex-Attorney General William Barr has repeatedly questioned publicly what his former boss was doing with classified material. There has been speculation that Trump might want such material as keepsakes or as some kind of leverage in a future business deal. But none of this is informed given the secrecy sounding the DOJ's investigation. […]
Link: https://www.cnn.com/2022/09/08/politics/republicans-trump-national-security-implications-classified-docs

Keyword-based
Summarize the above article in 2-3 sentences focusing on William Barr.

GPT3-D2:
The article discusses how various Republicans have been defending Donald Trump in light of the recent revelations that he had classified documents at his Mar-a-Lago resort. While some have downplayed the significance of the situation, others have outright denied that there is any cause for concern. However, former Attorney General William Barr has repeatedly questioned Trump's actions, speculating that he may have been trying to use the classified information for personal gain.
CTRLSum: Sen. Marco Rubio downplays the seriousness of the Department of Justice's investigation. Ex-Attorney General William Barr has repeatedly questioned what his former boss was doing with classified material. Trump and his allies have filled the vacuum for answers with a smokescreen of unproven claims, including that Trump declassified the material.

Aspect-based
In the above article, who is a defendant or under investigation?

In the above article,
what is the defendant's reaction to charges, including admission of guilt, denial of charges, or explanations?

GPT3-D2:
The defendant's reaction to charges in the above article is denial of charges. Trump and his allies have filled the vacuum for answers with a smokescreen of unproven claims, including that Trump declassified the material or that it might have been planted by the FBI in order to persecute him.
CTRLSum: Republican defenders of Donald Trump won't or can't answer the questions that are at the root of the intrigue over why classified documents were at Mar-a-Lago and the troubling question of whether national security was put at risk. In the article, who is a defendant or under investigation? Incorrectly answers the query.
Does not directly answer query.
Summary captures aspect-relevant content. Figure 6: Comparison of keyword-and aspect-based summaries using GPT3-D2 and CTRLSum models. The GPT3-D2 prompt is shown on the left with the corresponding keyword or aspect bolded. For keyword-based summarization, the GPT3-D2 summary presents appropriate context before the keyword-specific information. However, for aspect-based summarization, it does not always generate factually correct summaries, as shown in the first aspect example. We observe that CTRLSum performs poorly for both these settings.
interference in 2016 elections are keyword controls whereas charges against the defendants is a higherlevel aspect that can serve as the query for any news article discussing a lawsuit or investigation.

Qualitative Analysis
Baseline Model for comparison We use the recently proposed CTRLSum (He et al., 2020), a finetuned BART model, as our baseline. It can be flexibly adapted for both keyword-and aspect-based settings by including a prompt as additional input to the encoder. We use the prompt template recommended in the original paper. 13 Control Units For the keyword-focused setting, we use named entities extracted from the input article as the control units. For aspect-focused summarization, we directly use the aspects introduced in the guided summarization task from TAC 2011. 14 It defined 5 broad categories of newswire articles, 13 Trained model publicly released at: https://github. com/salesforce/ctrl-sum.
14 https://tac.nist.gov/2011/Summarization/ Guided-Summ.2011.guidelines.html such as accidents and natural disasters, investigations and trial, etc., and multiple aspects for each category. For example, the "investigations and trials" category includes aspects such as "who is the defendant or under trial?", "who is investigating, prosecuting, judging?", and so on. Figure 6 shows examples of keyword-and aspect-focused summaries using GPT3-D2 and the baseline CTRLSum model. The keywords or aspects are highlighted in bold within the GPT3-D2 prompt displayed on the left.

Qualitative Analysis
In this example, representative of average GPT3-D2 quality, the keyword-focused GPT3-D2 summary first gives a brief overview of the article setting before providing keywordrelevant information. In contrast, the CTRLSum summary exhibits poor discourse structure and reads like a list of facts stapled together.
The figure also shows aspect-focused summaries for two aspects associated with the "investigations and trial" category most appropriate for the chosen article. We see mixed results here for GPT3-D2; it generates a factually incorrect summary for the first aspect, listing multiple people from the input article as defendants instead of only "Donald Trump". For the second aspect, it correctly maps the highlevel concept "defendant" to "Donald Trump" in the input article and generates the correct answer to the input query: "The defendant's reaction to charges in the above article is denial of charges".
On the other hand, CTRLSum fails to generate aspect-focused summaries for both cases. We believe that it struggles to align high-level concepts and explicit entities in the article due to a lack of such aspect-specific examples in its training data. Instead, it generates summaries focusing on lexically similar words, i.e. "defenders" for both cases.
Based off of GPT3-D2's promising keywordfocused summarization capabilities observed above, we next conduct a human study to systematically compare it against the CTRLSum baseline. We leave further explorations of aspect-based summarization to future work, given the mixed to poor results for both models at this task.

Human Study: Keyword-focused summarization
Task Setup Similar to Section 3, we design an A/B test to compare the two models. We use the same set of 100 CNN 15 articles as Section 3. We randomly extract 2 distinct named entities from each article. In the study interface, the annotator is shown the article-keyword pair and GPT3-D2 and CTRLSum summaries corresponding to it. They are asked to select the summary that best summarizes the input article while focusing on the given keyword. Exact task instructions are included in Appendix F. Again, we run this study using the Prolific platform. We recruit 60 participants to annotate the 100 articles; each article is annotated by 3 annotators which includes annotations for 2 separate keywords. Each annotator evaluates 5 articles. Figure 7 shows the distribution of annotator votes between the GPT3-D2 and CTRLSum models. Annotators show a clear preference for GPT3-D2. In fact, for nearly 70% of all articlekeyword pairs, GPT3-D2 is preferred over CTRLSum by a majority of the annotators. The main rationales given for this choice were better contextualization of keyword-related information and better coherence in GPT3-D2 summaries. 15 We run this study using only CNN articles as the baseline CTRLSum model is trained on CNN/DM.  Figure 7: Distribution of annotator votes for the keyword-focused summarization task. Annotators prefer GPT3-D2 summaries over CTRLSum for approximately 70% of all article-keyword pairs, showing unanimous preference more than half the time.

Results
Impact These results show that zero-shot GPT-3 models present a promising alternative to finetuned models for such specialized summarization tasks that can be easily described using textual prompts. One of the major drawbacks of fine-tuned models is that they are constrained by what data is available and how it can be transformed to create new task-specific training data. CTRLSum relied on the SQuAD question answering dataset (Rajpurkar et al., 2016) because the required "queries" or "questions" were unavailable at scale for summaries in standard summarization datasets. In contrast, zero-shot models are not constrained by the availability of task-specific data and can flexibly adapt to new tasks. Future research should focus on further exploring these capabilities and possible improvements on currently "unsolved" tasks such as aspect-based or plan-based summarization.

Discussion and Related Work
In This brings us to manual evaluation, generally considered to be the gold standard for generation evaluation. The majority of summarization research now reports results from a human study in addition to automatic metrics, but there is a general lack of consensus on what dimensions to evaluate, task design, and other factors (Hardy et al., 2019). This presents difficulties in conducting reliable and reproducible comparisons between systems (Karpinska et al., 2021), another factor contributing to the popularity of automatic metrics. Although recent efforts like GENIE (Khashabi et al., 2021) have taken steps to standardize manual evaluation protocols across systems, its annotation is not universally affordable and the quality is not strictly monitored. We hope that future work addresses these challenges and democratizes human evaluations.
The ultimate test of summarization systems is with actual users using the systems in practice. Jones (2007) discusses the need to align task formulations with actual applications scenarios ("purpose factors"). However, the research in text summarization until now has been constrained to certain problems or domains by the heavy dependence on large-scale training data: for example, producing a bullet-point summary of a news article has emerged as standard due to availability of data from CNN, not because it is shown to be the best way to present information. Now, the success of zero-shot models like GPT3-D2 can allow realistic use-cases to drive research in a more top-down way. We already show that GPT3-D2 improves upon prior keywordfocused summarization systems that were trained on artificially adapted training data. In future research, we are interested in tackling other real world use cases, such as update summarization and plan-or aspect-based summarization. Additionally, adapting GPT3-D2 to documents longer than the allowed context, or structured inputs such as tables, presents research challenges beyond the current capabilities of GPT-3 and would be interesting to study. 16

Conclusion
In this work, we performed the first systematic study comparing zero-shot GPT-3 and fine-tuned models at the news summarization task. We ana-

A Implementation Details
Prompts Used To generate GPT3-D2 summaries for all experiments in this paper, we use the standard prompt format outlined in Section 2. We set N = 3 for CNN and DailyMail, N = 2 for Newsroom, and N = 1 for XSum/BBC. For the latter, the prompt is slightly modified to "Summarize the above article briefly in 1 sentence." For T0, we use the following prompts: a) CNN/DM: "Summarize the article below in 3 to 4 sentences?", b) Newsroom: "Summarize the article below in 2 to 3 sentences?", and c) XSum/BBC: "Summarize the article below in 1 sentence?" Factuality Metrics In Section 4.2, we evaluated several recently proposed factuality metrics. We note that multiple versions have been released for some of these models in recent years. Here, we specify the versions used in our experiments to ensure reproducibility of results:  Figure 8: Correlation between summary length and annotator score (computed as the no. of "best summary" votes. For each example, plot the difference in length (x-axis) and annotator score (y-axis) between the GPT3-D2 summary and the next best system's summary.
1. QuestEval: We use version 0.2.4 of the questeval python package and report numbers using the precision-only setting.

DAE:
We use the updated version of the DAE model trained for document-level factuality. Latest code and model released at https://github.com/tagoyal/ factuality-datasets.
Keyword-based data For our keyword-based human study, we extracted two named entities per article, as discussed in Section 5. In practice, we constrained the first keyword to be lead-biased, i.e. it was extracted from the first three sentences of the article, and the second keyword was extracted from the remaining article. As CNN-based summarization models are generally lead-biased, this allowed us to benchmark models under both settings.
B Are annotator judgments of quality correlated with length?
In Section 3, results of the human study showed that annotators provide shorter length as one of the main reasons for selecting T0 summaries as the worst for the CNN dataset. Here, we investigate if the choice between GPT3-D2 and BRIO is similarly influenced by their length differences; GPT3-D2 summaries are on average 9 words longer.
The article is about four men who have completed a world record by playing a board game continuously for more than 85 hours.
Four men completed a world record by playing a board game continuously for more than 85 hours.
Eight defendants have been charged with murder after two men died in a car crash.
Eight people have been charged with murder after two men died in a car crash on a dual carriageway.

CNN-style 3 sentence summaries
The article discusses Russian president Vladimir Putin's speech on Victory Day, in which he blended history with the present and justified his war. It is speculated that Putin may use this day to escalate the conflict in Ukraine, but it is also noted that he has few successes in Ukraine to brag of. The question now is whether Putin will use this day to escalate the war in other ways.
Russian President Vladimir Putin's speech at Russia's Victory Day commemorations offered few clues on the direction of the conflict in Ukraine. UK defense chief Ben Wallace had suggested that Putin may use this historic day to escalate his so-called "special military operation" in Ukraine and declare an outright war. Putin used his speech to blend history with the present, banking on Russian nationalism on its most patriotic of holidays to justify his war.
To study this, we plot the difference in summary length against the difference in annotator score (measured as the no. of votes for a summarization system) between the best summarization system (GPT3-D2) and the next best system (BRIO for CNN and T0 for BBC). The resulting plot is shown in Figure 8. In general we observe low correlation between these; Pearson's ρ is 0.17 for CNN and .02 for the BBC dataset. These correlation values cannot solely explain the large differences in annotator judgments reported in the human study results of Section 3; additional quality factors must have influenced this choice. Anecdotally, we observe that the GPT summaries are slightly less information dense; our impression is that these contain a similar level of information content, but are easier to read and understand despite being a bit more verbose. Figure 9 shows examples comparing summaries from text-davinci-001 (GPT3-D1) to those from GPT3-D2. For BBC-style single sentence summaries, we observed that the two models generated very similar summaries with high content and lexical overlap. More variance is observed for CNNstyle summaries. In our anecdotal assessment, GPT3-D1 generated more detailed summaries while those from GPT3-D2 are less information dense.

Common prompt for all segments:
Summary for input segment 2: The article discusses how the current situation with Covid-19 cases rising again is similar to last year in some ways, but different in others. One big difference is that there are now vaccines available which decrease the risk of severe illness. The article advises that everyone should take precautions such as wearing masks and opening windows to let fresh air in.

Summary for input segment 3:
The article discusses the challenges of living through the pandemic and the importance of science and community. It also highlights the need for patience, hope, and humility as we enter the next phase of the pandemic.

Summarize the above article briefly in 2-3 sentences.
Figure 10: Illustrative example of GPT3-D2 summary of a long source article generated using the segment-thensummarize pipeline.

D Human study and API costs
At the time of running our experiments, GPT-3 API's text-davinci-002 version was priced at $0.06 per 1K tokens. New pricing information is available at: https://openai.com/api/pricing/.
In our experiments, we generated around 2600 GPT3-D2 summaries across all experiments in Section 3 (human study), Section 4 (evaluation of metrics) and Section 5 (keyword-based human study). We spent a total of approximately $150 on API requests.
For the human study, we paid participants $4 per task (each task involved annotation for 5 articles). On average, this translated to $11/hr of work. The combined cost for the generic summarization (Section 3) and the keyword-based summarization (Section 5) studies was $1020, including platform costs and bonus payments.

E Long document summarization using GPT3-D2
Summarization of long documents has attracted significant interest in recent years (Cohan et al., 2018;Kryscinski et al., 2021). Here, we study how a naive zero-shot application of GPT-3 performs at long-document summarization. First, we extract text from a long input article from the CNN website. 17 Next, we follow the commonly used segment-then-summarize procedure from prior work (Zhao et al., 2020;Zhang et al., 17 Article link: https://www.cnn.com/2021/09/07/ opinions/covid-19-good-and-bad-news-ranney/ index.html 2022). We divide the input article into 3 disjoint segments, summarize each segment separately and concatenate these outputs to form the final summary. Figure 10 shows the prompt used and the generated summaries for each segment. While individual segment summaries are high quality, we can see that the concatenated summary is not coherent and includes repeated "introductory" sentences outlining similar content. Related to this, it also does not cover all important aspects of the input article as a majority of its 'length budget' is spent on a high-level overview. We also observed that the generated summaries for long documents often focus on less unimportant parts of the document, e.g. "...everyone should take the precaution of ... opening windows to let the fresh air in" in the illustrated example. This is, in part, due to the segmentation of the input article: GPT3-D2 still exhibits some lead bias and treats the beginning of the input segment as more salient. Therefore, the exact segmentation of the article also dictates the quality of the final summary, and cannot be readily fixed by altering the prompt.
These observations show that while GPT3-D2 produces superior segment-level summaries, it is more difficult to adapt it to "non-natural" text inputs without fine-tuning. Therefore, techniques that have shown promising results for fine-tuned models, e.g. segment-then-summarize or extractthen-abstract (Zhang et al., 2021) approaches, are not as effective when directly applied with zeroshot models.

F Task Instructions
Task instructions provided to crowd annotators for the generic summarization task setting are shown in Figure 14 and those for the keyword-based setting are shown in Figure 15.

G Examples of generated summaries
We show examples of generated summaries for articles for generic summarization for CNN-2022 andBBC-2022 in Figures 11 and 12. It includes summaries from the 3 different summarization models evaluated in the human study in Section 3.
Examples of keyword-focused summaries are shown in Figure 13 for CNN. It includes summaries generated by GPT3-D2 and CTRLSum models.
(CNN) Toronto Police fatally shot a man who was seen carrying a firearm near three schools in the Scarborough area Thursday, police said. Officers responded to reports of a man carrying a firearm and "there was an interaction" between officers and the man, according to the Special Investigations Unit in the Canadian province of Ontario. At some point during the interaction, two officers shot at the suspect, hitting him at least once, police said. The suspect was pronounced dead at the scene. At least four schools in the area were under lockdown Thursday as authorities tried to determine the extent of the threat, according to a series of tweets from the Toronto District School Board. The incident comes as the US is grappling with the shock of two recent mass shootings: its deadliest school shooting in nearly a decade in Uvalde, Texas, and a racist shooting at a supermarket in Buffalo, New York. "We certainly understand the trauma and how traumatic this must have been for staff, students and parents given the two recent events that have happened in the United States," said Toronto Police Chief James Ramer at a news briefing after the incident, referencing the Uvalde and Buffalo shootings. He added that he understands the community's concern as the armed suspected was very close to schools. The schools under lockdown included William G. Davis Junior Public School, Joseph Howe Senior Public School, Charlottetown Junior Public School and Sir Oliver Mowat Collegiate Institute, according to a tweet from the school board. "I was really scared, I just started praying," one student told CNN affiliate CTV. Ontario's Special Investigations Unit is now investigating the fatal shooting, according to a news release. "Four investigators and three forensic investigators have been assigned to the case," the release said. Ontario premier Doug Ford tweeted his thanks to police and emergency services for their "quick action." "Thank you to police and emergency services for your quick action today in Scarborough," he said. "We're extremely grateful for everything you do to keep our communities safe." BRIO: Suspect was seen carrying a firearm near three schools in the Scarborough area of Toronto, police say. At least four schools are under lockdown in the area. The incident comes as the U.S. is reeling from two recent mass shootings. The suspect was shot and killed by police. T0: NEW: "I was really scared, I just started praying," one student says. Shooting comes after two mass shootings in the United States. The gunman was spotted near three schools in the Scarborough area.

GPT3-D2:
The article discusses a police shooting that took place near several schools in Scarborough, Canada. The suspect was shot and killed by officers after an interaction during which the suspect was seen carrying a firearm. The incident is being investigated by the Special Investigations Unit.
(CNN) The owner of seven nursing homes across Louisiana that evacuated residents to a warehouse as Hurricane Ida approached last year has been indicted on felony charges after seven residents died at the temporary shelter, officials said. Bob Glynn Dean was arrested and charged with eight felony counts of cruelty to persons with infirmities, five felony counts of Medicaid fraud and two felony counts of obstruction of justice, according to a Wednesday news release from Louisiana Attorney General Jeff Landry. Five of the seven deaths at the warehouse shelter were considered storm-related, state health officials said. In total, more than 800 residents were taken to the facility ahead of the storm. A joint investigation by the attorney general's Medicaid Fraud Control Unit (MFCU) and the Louisiana Bureau of Investigations (LBI) "revealed Dean refused to move his residents out of the warehouse following Hurricane Ida, billed Medicaid for dates his residents were not receiving proper care, and engaged in conduct intended to intimidate or obstruct public health officials and law enforcement," the AG's news release said. Dean's attorney, John McClindon, told CNN Wednesday Dean plans to plead not guilty to all charges he's facing and said Dean's mental health will factor into the case. "I don't think it's any secret that Bob's mental health is going to be an issue in this case," McClindon said. "Bob clearly has some cognitive impairments and did on the day of this incident." McClindon said Landry informed him of an arrest warrant for Dean Monday and his client flew from Georgia to self-surrender. He made an initial court appearance in Louisiana Wednesday and was released on $350,000 bond, McClindon said. McClindon called the charges "very defensible" and said "the evidence will bear out eventually." Residents kept in 'unsafe, unsanitary, and unhealthy' conditions, officials said The nursing home residents were taken to the warehouse in Independence, about 57 miles east of Baton Rouge, ahead of Hurricane Ida's landfall on August 29. The state health department said it soon started to hear about deteriorating conditions at the warehouse. CNN obtained the logs of 61 calls from the warehouse to 911 operators. At least 30 of the calls asked for assistance with medical episodes before and after landfall, including calls for seizures, stopped breathing, and one instance in which a caller says a diabetic patient needed transport because they had "not eaten due to them having no more supplies." "Let's be clear; there is no emergency-preparedness plan that allows for residents to be kept in such an unsafe, unsanitary, and unhealthy condition," Stephen Russo, director of legal, audit and regulatory affairs for the health department, said last year. "The lack of adequate care for these residents is inhumane, and goes against the rules, regulations, and applicable statutes." The seven facilities involved had their licenses revoked and cannot repatriate or admit residents, officials said at the time. The homes also had their Medicaid provider agreements terminated, the health department said. The Attorney General's Office investigation is ongoing and additional legal action may be filed in the future, the Wednesday release said. The next court date for Dean has not been set, but McClindon said it will most likely happen in the next 60 days.

BRIO:
Owner of seven nursing homes evacuated residents to a warehouse as Hurricane Ida approached last year. Seven residents died at the temporary shelter. Bob Glynn Dean's attorney says he plans to plead not guilty to all charges. Five of the seven deaths at the warehouse shelter were considered storm-related, officials say.
T0: Bob Glynn Dean faces eight felony counts of cruelty to persons with infirmities. Dean's attorney says his client's mental health will be an issue in the case. Seven nursing home residents died at the warehouse shelter during Hurricane Ida.

GPT3-D2:
The owner of seven nursing homes in Louisiana has been indicted on felony charges after seven residents died at a temporary shelter during Hurricane Ida. The owner, Bob Glynn Dean, is facing eight felony counts of cruelty to persons with infirmities, five felony counts of Medicaid fraud, and two felony counts of obstruction of justice. Dean's attorney says Dean's mental health will be a factor in the case.
(CNN) Global leaders and defense officials had spent weeks speculating what Russian President Vladimir Putin might reveal about his Ukraine plans in a speech at Russia's Victory Day commemorations Monday. They'll have to keep guessing --the leader offered few clues on the direction of the conflict. UK defense chief Ben Wallace had suggested that Putin may use this historic day to escalate his so-called "special military operation" in Ukraine and declare an outright war. Even if that had been Putin's plan, he was unlikely to follow through after Wallace's comments, not wanting to appear to his Western foes as such an easy nut to crack. Instead, the Russian president used his speech to blend history with the present, banking on Russian nationalism on its most patriotic of holidays to justify his war. In his reverence for Soviet war heroes who helped defeat Nazi Germany in World War II --the reason Russia celebrates Victory Day --Putin referred to new Nazi threats in Ukraine, repeating his baseless justification for the invasion as an operation to "denazify" the nation. In reference to the threat of NATO troops in Europe, Putin said: "Everything indicated that a clash with neo-Nazis, Banderites [Ukrainian nationalists], on whom the United States and their younger partners counted on, would be inevitable." "Danger was increasing every day. Russia repelled this aggression in a preventative way. This was the only correct decision, and it was a timely decision. The decision of an independent, sovereign and powerful nation," he said. Putin had few other options than to use his speech to keep selling his war to his own people. He has so few successes in Ukraine to brag of, after all. All he can do now is to keep Russians on his side as they suffer the economic hardship of crippling sanctions and international isolationism. The question now is whether Putin will use this day --or this week even --to escalate the war in other ways. There are growing concerns that Russian forces will turn again to standoff weapons --aerial strikes and long-range missiles, for example --that can be fired from afar, as they so often do when they are on the backfoot. That's worrying, as those attacks are indiscriminate and tend to cause huge civilian tolls. A bombing of a school in Luhansk, eastern Ukraine, that is feared to have killed at least 60 sheltering people over the weekend is just one example. After Russia's failure to take territory in Ukraine's north and around the capital, Kyiv, it's struggling even in the east and south, where it has had a presence through pro-Russian rebels for years. The possibility that Russia may win nothing, or very little, in Ukraine is real. Whether anything changes this Victory Day or not, a new chapter in the war will inevitably need writing soon.

BRIO: Russian President Vladimir Putin gave a speech at
Russia's Victory Day commemorations on Monday. Peter Bergen says Putin gave few clues about his Ukraine plans in his speech. He says Putin used his speech to blend history with the present to justify his war in Ukraine. The question now is whether Putin will escalate the war in other ways, he says.

T0:
Vladimir Putin offered few clues about the direction of the conflict in Ukraine. Putin used his speech to blend history and the present to justify his war.

GPT3-D2:
The article discusses Russian president Vladimir Putin's speech on Victory Day, in which he blended history with the present and justified his war. It is speculated that Putin may use this day to escalate the conflict in Ukraine, but it is also noted that he has few successes in Ukraine to brag of. The question now is whether Putin will use this day to escalate the war in other ways.

Input Article
Generated Summaries Figure 11: Examples of generated summaries for the CNN-2022 dataset using 3 different summarization systems.
Four men have completed what they hope will be a world record by playing a board game continuously for more than 85 hours. The foursome, of Gloucester, completed their mammoth challenge on Monday night with just a couple of hours of sleep. Lea Poole, Dale Poole, Adam Bircher and Luke de Witt Vine played the game Dune, watched by supporters and an online audience. The world record attempt was carried out in aid of Alzheimer's Research UK. The rules of the challenge stated the men were allowed to accumulate five minutes respite for every hour they played. They had to beat the previous world record, held by a team in The Netherlands, which played a board game for 80 hours in January 2017. Guinness World Records stipulated they would need to play for at least five more hours to be considered for a world record. The men's attempt has yet to be ratified. Alzheimer's Research UK was selected to benefit from the challenge as Lea and Dale Poole's father suffers from the condition. Dale Poole said: "It's a bit of a rollercoaster really. It's very flattering. People have been sending in their support and donations and it's really humbling." The team were not allowed full meals, but just snacks throughout the attempt, which included 79 games in total. "You can be within one hour quite wide awake and alert, and by sitting down and not having to think for a few seconds, that's it, you're now tired and there's no way of breaking that weird cycle," said Lea Poole. The men completed the challenge at 23:00 GMT on Monday but decided to carry on and finish the game they were on, sipping on champagne to celebrate. Mr Burcher said he was "tired, beyond measure" but "really quite satisfied and pleased with how far we've come". "If you'd told me last year I'd be doing something like this I would have called you crazy," he added. Mr De Witt Vine said it had been a "difficult and trying experience but I'm glad we've done it." The group have raised almost £1,000 for Alzheimer's UK, as well as a smaller amount for Herefordshire Mind, a mental health charity.

BRIO:
A team of four men have set a new world record by playing a board game for 90 hours.
T0: Four men have broken the world record for the longest time spent playing a board game.

GPT3-D2:
The article is about four men who have completed a world record by playing a board game continuously for more than 85 hours.
A yellow warning for snow and ice has been issued for much of the North East. The Met Office warning is due to come into force at 20:00 BST on Wednesday and will run until 10:00 on Thursday. There will probably be icy patches on untreated roads and paths with higher elevation roads will be "affected by snowfall", the Met Office said. Motorists are being urged to take care. The cold snap comes days after the region basked in warm sun and highs of 20C (68F). The weather warning will cover Northumberland, County Durham, Tyneside, Darlington and Teesside. The wintry weather is expected to last until the weekend when slightly warmer temperatures will come in from the west, bringing unsettled conditions. BRIO: Parts of the UK are set to be hit by icy conditions, forecasters have warned.

T0:
The wintry weather is set to return to the North East of England on Wednesday night into Thursday morning.

GPT3-D2:
The Met Office has issued a warning for snow and ice in the North East, which is expected to last until the weekend.
A man standing trial for the murder of his partner's three-year-old son had convictions for common assault, burglary and possession of heroin with intent to supply, a jury has heard. Kemarni Watson Darby suffered fatal abdominal injuries at his mother's flat where he lived with Nathaniel Pope. Alicia Watson, 30, from West Bromwich, is also on trial for murder. Mr Pope, 32, from Wolverhampton, and Miss Watson both deny that charge and another of child cruelty. During his cross-examination, Miss Watson's QC, Charles Sherrard, told the court Mr Pope had been sentenced to four months' imprisonment for common assault in May 2011, 12 months for burglary in September 2011, and 20 months for possessing heroin with intent to supply in July 2013. Mr Pope conceded he had "just told some of the truth" about himself in evidence heard last week. Mr Sherrard referred to another arrest in March 2021. He said: "On the journey to Oldbury custody suite and on arrival at Oldbury custody suite you were asked by police officers several times if you had anything secreted on your person that you needed to present to them. "Each time you said to them 'no'. The custody suite police officers were authorised to conduct a strip search of you. "Two police officers then took you into a room and asked again if you had any drugs secreted on your person. Once again, you said 'no'. "A rock of 9.16 grams of crack cocaine was found in a white tissue secreted between your buttocks." Mr Sherrard also referred to Mr Pope's evidence last week that he was "merely a recreational user of cannabis". He asked: "(In) 2013 you're a dealer of heroin and in 2021 it's crack cocaine. Are you saying that in between we should read into it there would have been nothing of the sort?" Mr Pope answered: "There was nothing of the sort." The trial continues.

BRIO:
A man accused of murdering a toddler had a string of convictions, Wolverhampton Crown Court has heard.
T0: A man accused of murdering a toddler in Wolverhampton had convictions for assault and drugs offences, a court has heard.

GPT3-D2:
The man on trial for the murder of his partner's 3-year-old son had a history of convictions for assault, burglary, and drug dealing. Figure 12: Examples of generated summaries for the BBC-2022 dataset using 3 different summarization systems.