{'fraction_non_alphanumeric': 0.028160200250312892, 'fraction_numerical': 0.011333611458767904, 'mean_word_length': 4.948304383788255, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Extractive Multi-Document Summaries Should Explicitly Not Contain Document-Specific Content

Unsupervised approaches to multi-document summarization consist of two steps: finding a content model of the documents to be summarized, and then generating a summary that best represents the most salient information of the documents. In this paper, we present a sentence selection objective for extractive summarization in which sentences are penalized for containing content that is specific to the documents they were extracted from. We modify an existing system, Hier-Sum (Haghighi & Vanderwende, 2009), to use our objective, which significantly outperforms the original HierSum in pairwise user evaluation. Additionally, our ROUGE scores advance the current state-of-the-art for both supervised and unsupervised systems with statistical significance.


Introduction
Multi-document summarization is the task of generating a single summary from a set of documents that are related to a single topic. Summaries should contain information that is relevant to the main ideas of the entire document set, and should not contain information that is too specific to any one document. For example, a summary of multiple news articles about the Star Wars movies could contain the words "Lucas "and "Jedi", but should not contain the name of a fan who was interviewed in one article. Most approaches to this problem generate summaries extractively, selecting whole or partial sentences from the original text, then attempting to piece them together in a coherent manner. Extracted text is se-lected based on its relevance to the main ideas of the document set. Summaries can be evaluated manually, or with automatic metrics such as ROUGE (Lin, 2004).
The use of structured probabilistic topic models has made it possible to represent document set content with increasing complexity (Daumé & Marcu, 2006;Tang et al., 2009;Celikyilmaz & Hakkani-Tur, 2010). Haghighi and Vanderwende (2009) demonstrated that these models can improve the quality of generic multi-document summaries over simpler surface models. Their most complex hierarchial model improves summary content by teasing out the words that are not general enough to represent the document set as a whole. Once those words are no longer included in the content word distribution, they are implicitly less likely to appear in the extracted summary as well. But this objective does not sufficiently keep document-specific content from appearing in multi-document summaries.
In this paper, we present a selection objective that explicitly excludes document-specific content. We re-implement the HIERSUM system from Haghighi and Vanderwende (2009), and show that using our objective dramatically improves the content of extracted summaries.

Modeling Content
The easiest way to model document content is to find a probability distribution of all unigrams that appear in the original documents. The highest frequency words (after removing stop words) have a high likelihood of appearing in human-authored summaries (Nenkova & Vanderwende, 2005). However, the raw Figure 1: The graphical model for HIERSUM (Haghighi & Vanderwende, 2009). unigram distribution may contain words that appear frequently in one document, but do not reflect the content of the document set as a whole.
Probabilistic topic models provide a more principled approach to finding a distribution of content words. This idea was first presented by Daumé and Marcu (2006) for their BAYESUM system for query-focused summarization, and later adapted for non-query summarization in the TOPICSUM system by Haghighi and Vanderwende (2009). 1 In these systems, each word from the original documents is drawn from one of three vocabulary distributions. The first, φ b , is the background distribution of general English words. The second, φ d , contains vocabulary that is specific to that one document. And the third, φ c , is the distribution of content words for that document set, and contains relevant words that should appear in the generated summary.
HIERSUM (Haghighi & Vanderwende, 2009) adds more structure to TOPICSUM by further splitting the content distribution into multiple sub-topics. The content words in each sentence can be generated by either the general content topic or the content sub-topic for that sentence, and the words from the general content distribution are considered when building the summary. 1 The original BAYESUM can also be used without a query, in which case, BAYESUM and TOPICSUM are the exact same model.

KL Selection
The KL-divergence between two unigram word distributions P and Q is given by KL(P ||Q) = w P (w) log P (w) Q(w) . This quantity is used for summary sentence selection in several systems including Lerman and McDonald (2009) and Haghighi and Vanderwende (2009), and was used as a feature in the discrimitive sentence ranking of Daumé and Marcu (2006).
TOPICSUM and HIERSUM use the following KL objective, which finds S * , the summary that minimizes the KL-divergence between the estimated content distribution φ c and the summary word distribution P S : A greedy approximation is used to find S * . Starting with an empty summary, sentences are greedily added to the summary one at a time until the summary has reached the maximum word limit, L. The values of P S are smoothed uniformly in order to ensure finite values of KL(φ c ||P S ).

Why Document-Specific Words are a Problem
The KL selection objective effectively ensures the presence of highly weighted content words in the generated summary. But it is asymmetric in that it allows a high proportion of words in the summary to be words that appear infrequently, or not at all, in the content word distribution. This asymmetry is the reason why the KL selection metric does not sufficiently keep document-specific words out of the generated summary.
Consider what happens when a document-specific word is included in summary S. Assume that the word w i does not appear (has zero probability) in the content word distribution φ c , but does appear in the document-specific distribution φ d for document d. Then w i appearing in S has very little impact There will be a slight impact because the presence of the word w i in S will cause the probability of other words in the summary to be sligntly smaller. But in a summary of length 250 words (the length used for the DUC summarization task) the difference is negligible.
The reason why we do not simply substitute a symmetrical metric for comparing distributions (e.g., Information Radius) is because we want the selection objective to disprefer only document-specific words. Specifically, the selection objective should not disprefer background English vocabulary.

KL(c)-KL(d) Selection
In contrast to the KL selection objective, our objective measures the similarity of both content and document-specific word distributions to the extracted summary sentences. We combine these measures linearly: Our objective can be understood in comparison to the MMR criterion by (Carbonell & Goldstein, 1998), which also utilizes a linear metric in order to maximize informativeness of summaries while minimizing some unwanted quality of the extracted sentences (in their case, redundancy). In contrast, our criterion utilizes information about what kind of information should not be included in the summary, which to our knowledge has not been done in previous summarization systems. 2 For comparison to the previous KL objective, we also use a greedy approximation for S * . However, because we are extracting sentences from many documents, the distribution φ d is actually several distributions, a separate distribution for each document in the document set. The implementation we used in our experiments is that, as we consider a sentence s to be added to the previously selected sentences S, we set φ d to be the document-specific distribution of the document that s has been extracted from. So each time we add a sentence to the summary, we find the sentence that minimizes KL(φ c ||P S∪s ) − KL(φ d(s) ||P S∪s ). Another implementation we tried was combining all of the φ d distributions into one distribution, but we did not notice any difference in the extracted summaries.

Data
We developed our sentence selection objective using data from the Document Understanding Conference 3 (DUC) 2006 summarization task, and used data from DUC 2007 task for evaluations. In these tasks, the system is given a set of 25 news articles related to an event or topic, and needs to generate a summary of under 250 words from those documents. 4 For each document set, four humanauthored summaries are provided for use with evaluations. The DUC 2006 data has 50 document sets, and the DUC 2007 data has 45 document sets.

Automatic Evaluation
Systems are automatically evalatued using ROUGE (Lin, 2004), which has good correlation with human judgments of summary content. ROUGE compares n-gram recall between system-generated summaries, and human-authored reference summaries. The first two metrics we compare are unigram and bigram recall, R-1 and R-2, respectively. The last metric, R-SU4, measures recall of skip-4 bigrams, which may skip one or two words in between the two words to be measured. We set ROUGE to stem both the system and reference summaries, scale our results by 10 2 and present scores with and without stopwords removed.
The ROUGE scores of the original HIERSUM system are given in the first row of table 1, followed by the scores of HIERSUM using our KL(c-d) selection. The KL(c-d) selection outperforms the KL selection in each of the ROUGE metrics shown. In fact, these results are statistically significant over the baseline KL selection for all but the unigram metrics (R-1 with and without stopwords). These results show that our KL(c-d) selection yields significant improvements in terms of ROUGE performance, since having fewer irrelevant words in the summaries leaves room for words that are more relevant to the content topic, and therefore more likely to appear in the reference summaries.
The last two rows of  Table 1: ROUGE scores on the DUC 2007 document sets. The first two rows compare the results of the unigram HIERSUM system with its original and our improved selection metrics. Bolded scores represent where our system has a significant improvement over the orignal HIERSUM. For further comparison, the last two rows show the ROUGE scores of two other state-of-the-art multi-document summarization systems (Toutanova et al., 2007;Celikyilmaz & Hakkani-Tur, 2010). See section 6.2 for more details.
marization systems. Both of these systems select sentences discriminatively on many features in order to maximize ROUGE scores. The first, PYTHY (Toutanova et al., 2007), trains on dozens of sentence-level features, such as n-gram and skipgram frequency, named entities, sentence length and position, and also utilizes sentence compression. The second, HYBHSUM (Celikyilmaz & Hakkani-Tur, 2010), uses a nested Chinese restaurant process (Blei et al., 2004) to model a hierarchical content distribution with more complexity than HIERSUM, and uses a regression model to predict scores for new sentences.
For both of these systems, our summaries are significantly better for R-2 and R-SU4 without stopwords, and comparable in all other metrics. 5 These results show that our selection objective can make a simple unsupervised model competitive with more complicated supervised models.

Manual Evaluation
For manual evaluation, we performed a pairwise comparison of summaries generated by HIERSUM with both the original and our modified sentence selection objective. Users were given the two summaries to compare, plus a human-generated reference summary. The order that the summaries appeared in was random. We asked users to select which summary was better for the following ques- 5 Haghighi and Vanderwende (2009)   Q1 Which was better in terms of overall content?
Q2 Which summary had less repetition?
Q3 Which summary was more coherent?
Q4 Which summary had better focus?
We took 87 pairwise preferences from participants over Mechanical Turk. 7 The results of our evaluation are shown in table 2. For all attributes, our criterion performs better than the original HIERSUM selection criterion, and our results for Q1 and Q3 are significantly better as determined by Fisher sign test (two-tailed P value < 0.01).
These results confirm that our objective noticably improves the content of extractive summaries by selecting sentences that contain less document-specific information. This leaves more room in the summary for content that is relevant to the main idea of the document set (Q1) and keeps out content that is not relevant (Q4). Additionally, although neither criterion explicitly addresses coherence, we found that a significant proportion of users found our summaries to be more coherent (Q3). We believe this may be the case because the presence of document-specific information can distract from the main ideas of the summary, and make it less likely that the extracted sentences will flow together.
There is no immediate explanation for why users found our our summaries less repetitive (Q2), since if anything the narrowing of topics due to the negative KL(φ d ||P S ) term should make for more repetition. We currently hypothesize that the improved score is simply a spillover from the general improvement in document quality.

Conclusion
We have described a new objective for sentence selection in extractive multi-document summarization, which is different in that it explicitly gives negative weight to sentences that contain document-specific words. Our objective significantly improves the performance of an existing summarization system, and improves on current best ROUGE scores with significance.
We have observed that while the content in our extracted summaries is often comparable to the content in human-written summaries, the extracted summaries are still far weaker in terms of coherence and repetition. Even though our objective significantly improves coherence, more sophisticated methods of decoding are still needed to produce readable summaries. These problems could be addressed through further refinement of the selection objective, through simplification or compression of selected sentences, and through improving the coherence of generated summaries.