{'fraction_non_alphanumeric': 0.036510659152168584, 'fraction_numerical': 0.01037327452421792, 'mean_word_length': 4.505395683453237, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Scaling Up Query-Focused Summarization to Meet Open-Domain Question Answering

Query-focused summarization (QFS) requires generating a textual summary given a query using a set of relevant documents. However, in practice, such relevant documents are not readily available but should be first retrieved from a document collection. Therefore, we show how to extend this task to make it more realistic. Thereby the task setup also resembles the settings of the open-domain question answering task, where the answer is a summary of the top-retrieved documents. To address this extended task, we combine passage retrieval with text generation to produce the summary of the retrieved passages given the input query. We demonstrate the first evaluation results on the proposed task and show that a few samples are sufficient to fine-tune a large generative model with retrieved passages.


Introduction
Query-focused summarization (QFS) aims to produce a multi-sentence summary from a set of topic-related documents to answer a given query. One of the main challenges of this task is the dataset construction. A typical pipeline is for several professional assessors to first read candidate documents given a topic, then design a query reflecting the information-seeking needs and choose a subset of 25-50 documents relevant to this query. Finally, they write reference summaries based on the selected documents to answer the query [2]. Obviously, this labor-intensive work is very time-consuming. Therefore, all existing QFS datasets(e.g., DUC [2]) do not contain enough samples for training the data-hungry neural models. They can be used only for fine-tuning and evaluation.
Another notable limitation of the existing QFS task setup is that the relevant documents for the given query are assumed to be readily available, which is clearly not the case in practice. In a more realistic scenario, the relevant documents should be first retrieved from a large document collection, e.g., the Web, and after that summarised to answer the query. It is important to consider the end-to-end task to be able to analyse and mitigate the errors propagating from the retrieval phase.
We propose to extend the standard QFS task by replacing the set of relevant documents with a large-scale document collection. We refer to this new task setup as open-domain QFS. Such a modification in the task setup also makes it more similar to the open-domain question answering (QA) task, where a natural language question should be answered using a large document collection [1].
Most of the datasets for open-domain QA, such as Natural Questions [10], assume that the answer is a single contiguous span that can be extracted from a single relevant document (extractive QA). Reusing the QFS datasets in the open-domain setting allows us to obtain high-quality data for abstractive QA, in which the answer to the given question is a summary of multiple relevant documents.
Due to the low resource setting, the previous work on QFS relied mainly on unsupervised approaches to extract sentences to produce the output summary [6,13,16]. With the rising popularity of the pre-trained language models, the more recent studies [11,20,21] also explored weak/distant supervision from the QA and single-document summarization tasks. Such approaches incorporate sentence selector trained on large QA datasets, such as SQUAD 2.0 [19] to select relevant sentences, or train a summarization model on large non-query-focused summarization datasets, such as CNN-DM [7].
In contrast to all previous work, we explore the potential of the retrievergenerator framework that was recently proposed in the context of the opendomain QA task [5,4] to tackle the task of open-domain QFS, which we introduced in this work. In this framework, the retriever is set to output the top-k relevant passages for a given query. Then, the generator uses the encoder-decoder architecture, which takes the query and the passages retrieved at the previous stage as input, to produce the output summary.
Our main contributions are twofold: (i) we are the first, to our knowledge, to extend the QFS task into the open-domain scenario, and (ii) show the results of the state-of-the-art model on this task that can be efficiently trained in a low-resource scenario.

Task Formulation
We briefly introduce the original QFS task and then describe our proposed extension to the open-domain QFS. Let the tuple (q, D, s) denote a sample in the QFS dataset (a cluster ), where q is a given query, D = {d 1 , . . . , d m } is a set of topic-related documents and s is the reference summary for the document set D and the query q. The goal is to produce the summary given the documents and the query as input: (q, D) → s. Note that D are selected manually based on q by human annotators and usually includes tens of documents.
We extend the original QFS task to the open-domain setting as (q, P) → s by replacing D with P, where P = {p 1 , . . . p n } are millions of passages processed from a large-scale document collection, i.e., n m . Our goal remains to generate the summary s with respect to the query q. However, different from the original QFS setup, there is no evidence that the given passages are relevant to the query. More importantly, the number of passages is so big that it is impossible to consider all of them, which calls for an effective information retrieval model to supplement the extended task.

Method
Inspired by the approach previously proposed for the long-form QA task [5], we adopt a retriever-generator framework to address the task of open-domain QFS introduced in the previous section.

Retriever
The goal of the retriever is to return the top-k most relevant passages given the input query. Our retriever is based on the DPR model [9], which consists of a query encoder E Q (·) and a passage encoder E P (·). The encoders map the input query q and passage p to dense vectors E Q (q) and E P (q) respectively. The similarity score between them is defined as the dot product of their corresponding vectors: Then, top-k passages P k = {p 1 , p 2 , . . . , p k } are retrieved based on the above scores. We use DPR that was pre-trained on KILT benchmark [17] by fine-tuning the BERT base model [3].

Generator
Our generator is based on the Fusion-in-Decoder (FID) approach, which is the recent encoder-decoder architecture proposed for open-domain QA datasets [8].
Following this work, we use the pre-trained T5 base [18] to initialize our generator. The generator takes the query q and the set of the top-k retrieved passages P k as input and produces the final summary. More specifically, every supporting passage p i from P k is concatenated with the query and used as input to the encoder independently. Afterwards, the encoded representations of all the retrieved passages are concatenated and used as input to the decoder. Let E G (q, p i ) denotes the output representation of the encoder E G (·) for the passage p i , then the decoder D G (·) generates a summary S word by word:  Table 1 for more detailed dataset statistics).

Experiments
We then complement the query-summaries pairs from DUC with three different passage collections. The first one ('WIKI') is the Wikipedia dump used in the KILT benchmark [17], which contains about 5.9 million Wiki documents. Following the previous work [9], we split each document into disjoint passages with 100 words maximum. This way we obtain about 21 million passages in total. However, we can not guarantee that 'WIKI' contains enough evidence to answer the queries from DUC since the reference summaries are based on the content of the original DUC documents. Therefore, we mix all the DUC documents from different clusters together to construct the second document collection ('DUC') using the same approach to split the document into the passages, which results in 32 thousand passages in total. Finally, we combine both collections to form the third collection ('MIX').
Following the previous work on DUC datasets [14], we use the data of the first two years as the training set and the third subset as the test set. We randomly select 10% from the training set for validation.

Experimental Setup
In this work, we name our model DPR+FID. For comparison, we also apply BM25 as another retrieval method to get retrieved passages, and it is called BM25+FID. Moreover, we implement the recent state-of art models in opendomain question answering, RAG [12], in which passage retrieval and answer generation can be learned jointly.
Since we have three document collections ('WIKI', 'DUC' and 'MIX'), we have variants of each model trained on the passages retrieved from the respective collection, e.g., DPR+FID (WIKI) denotes that we use the passages from 'WIKI' for training and test. For DUC, we use the passages from the first two years for training/validation and the passages from the third year for test.   We only train the FID model and set k for top-k retrieved passages as 50. For training, the learning rate and the batch size are set to 1e-4 and 1, respectively. All of our models are trained on a single TITAN-XP GPU. We set the maximum number of training epochs to 30, which took about 12 hours. For inference, We set the maximum length of a generated summary to 250.

Results
In this section, we compare the retrieval performance of BM25 versus DPR, and evaluate the quality of the summaries generated by all our models. We also investigate the impact of the number of retrieved passages on the summarization performance.
Retrieval Performance. To evaluate retrieval components BM25 and DPR, we treat the retrieved passages from the same cluster with their corresponding query as the ground truth. Following the related work on DPR [9], we use top-k accuracy as the retrieval metric for this task. Table 2 compares BM25 and DPR on DUC 2007 dataset using the passages from the 'DUC' collection. We can see that DPR performs much better than BM25. With the increase of the number of passages k, we see the decrease in accuracy for both models.
Summarization Performance. The results of generated summaries on the extended DUC 2007 datasets are shown in Table 3. For evaluation, we use ROUGE [15] as an automatic metric of summarization quality. Following previous work [20,21], we report F 1 for ROUGE-1 (R-1), ROUGE-2(R-2) and ROUGE-SU4 (R-SU4) with maximum length of 250 words. We observe that the performance of RAG-based models is relatively lower, which means that they can not be trained well on such a small dataset. In contrast, BM25+FID achieves the best performance among all models but only on the smallest collection (DUC). As the number of passages increase from thousands to millions in the MIX collection, DPR-based model shows an increase in performance in comparison with BM25.
Note that BM25+FID (DUC) outperforms DPR+FID (DUC), while DPR was previously shown to outperform BM25 on the passage retrieval task for the same collection in Table 2. This may indicate that using all passages from the cluster is not sufficiently accurate to evaluate the retrieval performance in this case since only a few passages actually contain the evidence required to address the given query.
In Figure 1, we report the change in ROUGE-SU4 of the DPR+FID model (WIKI) with respect to the change in the number of retrieved passages, i.e., sensitivity of the summarization model to the hyper-parameter k. We observe that increasing the number of passages from 10-30 brings an improvement. However, the performance slightly drops when the number of passages surpasses 30, i.e., the summarization model is not sufficiently robust to deal with the noisy input and retrieving more passages may hurt the summarization performance.

Conclusion
In this paper, we propose to extend the original QFS task setup into the opendomain setting and investigate the performance of the retriever-generator framework on the extended DUC datasets. We show that a large summarization model for the open-domain QFS task can be successfully trained with less than one hundred samples by incorporating the passages obtained from the retriever. We release the code of our experiments and the extended DUC dataset to make our results easier to reproduce.