{'fraction_non_alphanumeric': 0.03483928242770686, 'fraction_numerical': 0.014344950036268364, 'mean_word_length': 4.875826972010178, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 4, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Trade-off between Redundancy and Local Coherence in Summarization

Extractive summarization systems are known to produce poorly coherent and, if not accounted for, highly redundant text. In this work, we tackle the problem of summary redundancy in unsupervised extractive summarization of long, highly-redundant documents. For this, we leverage a psycholinguistic theory of human reading comprehension which directly models local coherence and redundancy. Implementing this theory, our system operates at the proposition level and exploits properties of human memory representations to rank similarly content units that are coherent and non-redundant, hence encouraging the extraction of less redundant final summaries. Because of the impact of the summary length on automatic measures, we control for it by formulating content selection as an optimization problem with soft constraints in the budget of information retrieved. Using summarization of scientific articles as a case study, extensive experiments demonstrate that the proposed systems extract consistently less redundant summaries across increasing levels of document redundancy, whilst maintaining comparable performance (in terms of relevancy and local coherence) against strong unsupervised baselines according to automated evaluations.


Introduction
Automatic single-document summarization is the task of reading a text document and presenting an end-user (be it a human user or a module down a processing pipeline) with a shorter text, the summary, that retains the gist of the information consumed in the document. Such a complex task can be divided in the following three general steps: (i) discretization of the information in the source document into semantic content units and building a representation of these units, (ii) selection of content units such that they are relevant with respect to the source document, non-redundant among themselves, and informative to the end-user; and finally, (iii) production of a summary text that is coherent and cohesive. From the many variations of the summarization task investigated in recent years (Litvak & Vanetik, 2017;Shapira et al., 2017;Narayan et al., 2019;Xiao & Carenini, 2019;Amplayo et al., 2021), abstractive and extractive summarization approaches continue to garner intense attention. These approaches differ mainly in how the summary text is synthesized. Whilst abstractive approaches generate a fluent text conditioned in the selected content, extractive approaches concatenate the selected units and present them as a summary.
Even though recent advances in machine learning brought promising results -mostly involving increasingly larger neural networks-in all stages of the summarization pipeline, core challenges such as redundancy during content selection (Xiao & Carenini, 2020;Jia, Cao, Fang, Zhou, Fang, Liu, & Wang, 2021) and coherence of produced summaries (Cao, Wei, Li, & Li, 2018;Sharma, Li, & Wang, 2019;Hua, Sreevatsa, & Wang, 2021) remain critically open (Teufel, 2016;Gatt & Krahmer, 2018). Notably, Xiao and Carenini (2020) reported that modern extractive summarization systems are prone to produce highly redundant excerpts when redundancy is not explicitly accounted for. The problem becomes particularly acute when the source document is highly redundant, i.e. information is repeated in many parts of the document. Some examples of highly redundant documents include scientific articles, and in general, long structured documents. Consider the example in Figure 1 showcasing how information is repeated across sections in a scientific article. Information redundancy is characteristic of the writing style in scientific literature: the 'Introduction' section is expected to lay down the research questions addressed in the paper, each of which will be elaborated upon in following sections, and the 'Conclusion' section (or equivalent) gathers insights and summarizes the answers to each research question. In this work, we tackle the problem of summarizing long, highly redundant documents, and address this gap by proposing unsupervised extractive summarization systems capable of consuming long documents in linear time and extract significantly less redundant summaries than strong baselines.
Nevertheless, the problem of redundancy is not exclusive to automatically extracted (or generated) summaries. In particular, Kintsch (1990) reported that cognitive processes in charge of deleting or fusing redundant information become more effective in human subjects as their summarization skills are honed throughout educational levels. In cognitive psychology, summarization as a task is often used as a method to investigate cognitive processes involved in text comprehension and production (Kintsch & van Dijk, 1978;Kintsch, 1990;Lehto, 1996;Kintsch & Walter Kintsch, 1998;Ushiro et al., 2013;Spirgel & Delaney, 2016). Such processes are in charge of generalizing, synthesizing, and coherently organizing content units. Comprehension, in turn, is modelled after psycholinguistic models of human reading comprehension (Kintsch & van Dijk, 1978;Kintsch, 1988) which provide rich and robust theoretical foundation on how content units are discretized and manipulated by cognitive processes. For this reason, cognitive models of comprehension have drawn attention of researchers in automatic summarization in recent years (Fang & Teufel, 2014;Zhang et al., 2016;Fang, 2019). However, these cognitively-inspired systems show the following limitations. First, they make extensive use of out-of-the-box NLP tools such as parsers, named entity extractors, co-reference resolution systems, as well as external resources like WordNet (Miller, 1995) and Wikipedia. Second, they do not exploit content representations obtained through comprehension simulation for meaningful sentence scoring, relying on frequency heuristics instead. Finally, many design choices were taken to better accommodate the processing of short argumentative (Fang, 2019) or narrative texts (Zhang et al., 2016). Hence, the application of these systems to other setups (e.g. longer documents, other knowledge Introduction wolf rayet ( wr ) stars are evolved , massive stars that are losing their mass rapidly through strong stellar winds ( conti 1976 ) . in this scenario , hot , massive ob stars are considered to be the wr precursors that lose their external layers via stellar winds , leaving exposed their he burning nuclei and h rich surfaces . . . [at radio frequencies , the excess of emission is associated with the contribution of the free thermal emission coming from the ionized and expanding envelope formed by the stellar wind]• . . . in this paper , we present [simultaneous , multi frequency observations of a sample of 13 wr stars using the vla at 4. 8 , 8.4 , and 22.5 ghz] , aimed at [disentangling the origin of their stellar wind radio emission through the analysis of their spectral index and time variability by comparison with previous observations] .
Observations we performed [radio observations of a sample of 13 wr stars] , listed in table 1 , [with the very large array ( vla )] of the national radio astronomy observatory ( nrao ) . . .  domains) becomes very difficult. We address these limitations by introducing two computational implementations of the Micro-Macro Structure theory (Kintsch & van Dijk, 1978) that require only a dependency parser and no external resources, and exploits properties of content representations in order to capture informative, non-redundant information from long documents.
Additionally, we give special attention to the problem of control of summary length as a way to compare systems in a fairer way. Recent studies (Sun et al., 2019;Schumann et al., 2020) have pointed out the susceptibility to summary length of automatic evaluation metrics such as ROUGE (Lin, 2004), and reported that most recent work unknowingly exploit length differences, e.g. obtaining higher ROUGE scores by extracting longer sentences. We address this gap by formulating content selection as an optimization problem subject to a soft constraint in the number of tokens. The resulting summaries present less variability in length, hence ensuring a fairer comparison among models in terms of automatic metrics.
In summary, we tackle the problem of redundancy reduction in unsupervised extractive summarization of long, redundant documents, taking as case study scientific articles collected from ArXiv and PubMed. Our contributions are the following. 1 First, we investigate the trade-off between redundancy and local coherence in extractive summaries. To this end, we introduce two summarization models that rank groups of coherent, non-redundant information similarly by simulating how humans organize content in short and long-term memory. Second, we argue that control over summary length is crucial for a fair comparison of systems and propose a sentence selection strategy that controls for summary length in tokens. Extensive experiments demonstrate that our proposed systems produce significantly less redundant summaries for highly redundant documents, whilst maintaining competitive performance against strong unsupervised baselines according to automatic evaluation of relevancy and local coherence. The rest of the paper is organized as follows. An overview of previous work is presented in Section 2, followed by a detailed description of the KvD theory in Section 3. Then, Section 4 presents our proposed models inspired on KvD and our content selector. Finally, Sections 5 and 6 describe our experimental setup and discuss our results, respectively.

Related Work
In this section we discuss previous efforts related to automatic summarization, both traditional and modern (neural based), how the problems of content selection and redundancy are being tackled, and how cognitive science has influenced automatic summarization and the field of natural language processing in general.

Summarization Approaches
Extractive approaches to summarization aim to first obtain a representation of content in a source document, then select a subset of content units, and finally present the concatenation as a summary. Early approaches represented and organized content in a document using semantic and discourse methods such as lexical chains (?; Silber & McCoy, 2002), latent semantic analysis (Gong & Liu, 2001;Hachey et al., 2006), coreference information (Baldwin & Morton, 1998;Steinberger et al., 2007), and rhetorical structure theory (Ono et al., 1994;Marcu, 1998). In particular, graph representations proved effective in encoding relations between content units such as discourse relations (Wolf & Gibson, 2004;Louis et al., 2010) and word co-occurrence statistics (Mihalcea & Tarau, 2004;Erkan & Radev, 2004). After obtaining a representation of a document, the selection of content units (usually sentences) is posed as a unit ranking problem or a sequence labeling problem in which each unit is labeled as 'select' or 'not select'. For this selection stage, machine learning approaches have proven effective at identifying summary-worthy units (i.e. relevant and informative) by leveraging manually-crafted features such as word frequency (Vanderwende et al., 2007;Nenkova et al., 2006), sentence length , and the presence of keywords of proper nouns (Kupiec et al., 1995;Jones, 2007).
More recently, summarization approaches rely instead on neural networks to obtain deep representations of content units by means of convolutional neural networks (Perez-Beltrachini et al., 2019;Narayan et al., 2019), recurrent neural networks (Narayan et al., 2018a(Narayan et al., , 2018bCheng & Lapata, 2016), Transformers (Song et al., 2019;Dong et al., 2019) and lately by leveraging large pretrained language models (Zheng & Lapata, 2019;Zhang et al., 2020). Building up on traditional methods, neural summarization models leverage discourse (Clarke & Lapata, 2010;Cohan et al., 2018), topical (Narayan et al., 2019), and graph representations (Bichi et al., 2021). Even though most research concentrates on summarization of middle-sized documents like news articles and Reddit posts (Völske, Potthast, Syed, & Stein, 2017), recent work has shifted attention to long document summarization and its challenges (Cohan et al., 2018;Sharma et al., 2019;Xiao & Carenini, 2019). Among recent efforts it is worth mentioning architectures tailored to consume longer inputs by reducing the time complexity of the attention mechanism (Beltagy et al., 2020;Wang et al., 2020;Huang et al., 2021) or leveraging the structure of the input document Narayan et al., 2020). The present work follows this line of research by introducing summarization systems capable of consuming long documents and extracting a summary in linear time w.r.t. the number of sentences, although with large constants. Note that the proposed systems do not employ neural networks during content representation or selection but instead operate over cognitively-inspired structures of propositions representing human memory.
Finally, of special interest to this work are unsupervised approaches to summarization, an area not explored as much as its supervised counterpart given the availability of large summarization datasets nowadays (Hermann et al., 2015;Cohan et al., 2018;Narayan et al., 2019). On the one hand, abstractive approaches mostly rely on autoencoding strategies in order to reconstruct summaries from a topic-rich latent space (Yang, Zhu, Gmyr, Zeng, Huang, & Darve, 2020), a sequence of discrete latent variables (Baziotis, Androutsopoulos, Konstas, & Potamianos, 2019), or from the mean representation of relevant content units (Chu & Liu, 2019). On the other hand, central to most extractive approaches is a weighted graph representation of the source document (Bichi et al., 2021) followed by sentence ranking based on node centrality, where edge weights are calculated by TF-IDF (Mihalcea & Tarau, 2004) or by finetuned, dedicated architectures (Zheng & Lapata, 2019). Our work differs from this line of research in two aspects. First, content is organized in tree and graph structures where nodes are modeled as propositions instead of sentences or words. However, content selection is still performed at the sentence level. Second, the proposed node scoring strategy exploits cognitively-grounded properties of human memory structures. We demonstrate through extensive experiments that this scoring strategy performs on par with strong centrality-based baselines w.r.t automatic metrics, whilst ranking more coherent and less-redundant content units higher.

Content selection, Redundancy, and Length Control
Modern approaches mostly employ neural end-to-end models (Cheng & Lapata, 2016;Lewis et al., 2020;Zhang et al., 2020), meaning that crucial intermediate steps such as content planning or selection are not explicitly modeled, in contrast to traditional summarization approaches (Jones, 1993;Carbonell & Goldstein, 1998;Nenkova et al., 2011;Lloret, 2012;Teufel, 2016). Recent efforts have demonstrated that accounting for planning helps dealing with coherence of final summaries (Goldfarb-Tarrant et al., 2020;Sharma et al., 2019;Hua et al., 2021), whereas explicit content selection modules can be tailored to tackle problems such as factuality (Cao et al., 2018;Maynez et al., 2020), coverage (Kedzie et al., 2018;Puduppully et al., 2019;Wiseman et al., 2017), and redundancy Jia et al., 2021;Bi et al., 2021). Specifically, production of low-redundant summaries has proven to be challenging, especially when the source document is highly redundant, such as scientific articles. In particular, Xiao and Carenini (2020) reported that, among the neural approaches that deal with redundancy at different stages of the summarization pipeline, the ones that account for it during content selection or summary generation proved to be more effective. In this work, we demonstrate that by simulating comprehension according to a cognitive model, non-redundant content can be scored together in each simulation cycle, hence encouraging the selection of non-redundant content.
Additionally, the need for controlling aspects of generated text such as length, style, and mentioned entity has been acknowledged lately in an effort to enable models to adapt to user input (Fan, Grangier, & Auli, 2018) or predefined user preferences (Song, Wang, Feng, & Liu, 2021). Most importantly, previous work (Sun et al., 2019;Schumann et al., 2020) pointed out how the lack of control in produced summaries led recent summarization systems to unwillingly exploit the susceptibility of automated metrics to summary length. In order to guarantee a fairer comparison between systems, we control for extracted summary length by posing content selection as knapsack optimization problem subject to a soft budget of tokens. The proposed selector is able to extract summaries in a much restricted length range, making comparison between models much more reliable.

Cognitive Models for NLP Tasks
Cognitive science and artificial intelligence have a history of influencing each other ever since the beginning of both fields. On the one hand, recent work in cognitive psychology -the field that aims to answer questions about how humans think-employed natural language processing (NLP) models to investigate aspects of human language comprehension. For instance, language models and statistical parsers were used to explain language processing difficulty (Sarti et al., 2021;Rathi, 2021;Meister et al., 2022) and incrementality (Merkx & Frank, 2021;Stanojević et al., 2021), syntactic agreement processes (Ryu & Lewis, 2021), brain representations of abstract and concrete concepts (Anderson et al., 2017;Ramakrishnan & Deniz, 2021), prediction of gaze behaviour (Hollenstein et al., 2021), among others. In clinical psychology, Transformer-based language models are being used to formulate cognitive models that better explain human emotions (Guo & Choi, 2021), comprehension deficit in aphasia subjects (Guo & Choi, 2021), and even improve suicidal prevention systems (MacAvaney et al., 2021).
On the other hand, tasks in NLP have benefited from cognitive science mainly in two aspects, the availability of datasets gathered during behavioral tests (Barrett et al., 2018;Mathias et al., 2021) and by leveraging cognitive theories for model design guidance. Firstly, eye tracking and brain activity data (captured by functional magnetic resonance imaging, fMRI, and electroencephalography, EEG) proved useful for a wide range of tasks such as sentiment analysis (Gu et al., 2014;Mishra et al., 2018), relation extraction (McGuire & Tomuro, 2021), name entity recognition , and text simplification (Klerke et al., 2016). Secondly, cognitive theories of text comprehension and production have guided model design for grammar induction and constituency parsing (Levy et al., 2008;Wintner, 2010), machine translation (Saini & Sahula, 2021), common-sense reasoning (Sap et al., 2020), and training strategies involving regularization (Wei et al., 2021) and curriculum learning (Xu et al., 2020). Moreover, these theories played key roles in the understanding catastrophic forgetting during fine-tuning of neural networks (Arora, Rahimi, & Baldwin, 2019), better analysis of what neural networks comprehend (Ettinger, 2020;Dunietz et al., 2020), and better evaluation of generated text (van Der Lee et al., 2019), In particular, recent work in automatic summarization sought to simulate cognitive processes as defined by the Micro-Macro Structure (KvD; Kintsch and van Dijk, 1978) and Construction-Integration theory (CI; Kintsch, 1988). These theories outline procedures to discretize content in propositions, build text representations that account for local and global coherence, and aim to explain how content is manipulated in short and long-term human memory. However, computational implementations proposed so far (Fang & Teufel, 2014;Zhang et al., 2016;Fang, 2019) show a heavy reliance on NLP tools such as parsers, entity extractors, coreference resolution systems, as well as external resources like WordNet (Miller, 1995), making their application in closed domains (e.g. scientific literature) very unreliable. Additionally, many design choices prevented these systems from exploiting properties of memory structures, retrieval processes, or manipulation of information at the right granularity level (e.g. ranking words instead of propositions as KvD states). In this work, we introduce a novel implementation of the KvD theory that operates at the propositional level, and propose two extractive summarization systems that have little dependence on NLP tools, requiring a dependency parser in Universal Dependency format (De Marneffe et al., 2021), making it possible to test these systems on other languages and domains. Moreover, our proposed systems better exploit memory structure properties and retrieval processes during reading simulation, which makes them capable of producing notoriously less redundant and more coherent summaries than strong baselines.

The KvD model of Human Memory
In this section, we describe the cognitive theory of reading comprehension proposed by Kintsch and van Dijk (1978), KvD, and specify how it is relevant to the task of summarization. The KvD theory aims to describe the cognitive processes involved in text or speech comprehension, and provides a principled way to make predictions about the content human subjects would be able to recall later.
According to KvD, discourse comprehension is performed at two levels, micro and macrolevel, and discourse is represented with a characteristic structure of content at each level. At the micro level, content structure is modeled after working memory -a type of shortterm memory-and KvD defines precise mechanisms that update and reinforce content in the structure. Content at this level is discretized in basic meaningful units by means of linguistic propositions. A proposition is denoted as predicate(arg 1 ,arg 2 ,...) where arg i is a syntactic argument of the predicate (e.g. argument to a transitive verb). As such, propositions can be interpreted as clauses or short sentences and hence provide more expressivity than words units during comprehension. The advantage of using propositions as content units goes beyond the amount of information it can pack. A proposition can be linked to another either syntactically or semantically, potentially building entire connected structures of propositions. According to KvD theory, working memory holds a coherent organization of content units by making sure that all units are connected e.g. in a connected tree. Hence, the resulting micro-structure models local coherence and cohesiveness of the text.
At the macro level, content structure represents the global organization of the text and its building is guided by the reader's goals in mind whilst performing the task. For instance, if the task is summarization, KvD defines macro-processes concerned with generalization, insertion of details from background knowledge, among others. In this work, we consider only the structures represented at the micro level and leverage them for the task of extractive summarization. Structures and processes at the macro level would require human-like reasoning and intuition and even though recent work on neuro-symbolic systems (Garcez & Lamb, 2020;Bengio, 2017) and common-sense reasoning (Speer et al., 2017;Bosselut et al., 2019) show a promising development path, we leave this path out of the scope of this work and for future work.

Memory Simulation at Micro Level
At the micro level, content is organized in a structure representing working memory, modeled as a tree of propositions called the memory tree. According to KvD, reading is carried out iteratively in memory cycles. In each cycle, only one new sentence is loaded to the working memory. Then, propositions are extracted and added to the current memory tree. The limits of memory capacity is modeled as a hard constraint in the number of propositions that will be preserved for the next cycle. Hence, the tree is pruned and some propositions are dropped or forgotten. However, if nodes cannot be attached to the tree in upcoming cycles, these forgotten nodes can be recalled and added to the tree, serving as linking ideas that preserve the local coherence represented in the current tree. 2 We now illustrate with an example how content units are captured, forgotten, and recalled during a KvD simulation of reading.
Consider the first three sentences of the introduction section of a biomedical article, along with its abstract, showed in Figure 2. At the beginning of each cycle, propositions are extracted from the incoming sentence and connected to the existing memory tree, obtaining trees (1a), (2a), and (3a). Two nodes will be connected if they share an argument. For instance, node 5 and 6 share the argument antioxidants. Then, the most relevant nodes are selected using KvD's leading edge strategy, a strategy that aims to keep the most general and most recent nodes. In this example, we set the memory limit to 5 selected propositions per cycle. The rest are pruned, obtaining trees (1b), (2b), and (3b). These pruned trees constitute the final product of each cycle and will be used for our content selection experiments.
Let us now analyze what kind of information KvD preserves, forgets or recalls and how it is done. Note that in cycle 1 the selected root is node 4, a proposition containing the main verb of the sentence. However, the root is changed at the beginning of cycle 2 to node 7 (nonenzimatic antioxidants, #7), reflecting the change in focus. Note that #7 is the only proposition mentioned in both sentences, hence serving as link to insert the other nodes. Now, the tree showcases clearly two ramifications of the current topic, namely that '#7 control a specific kind of molecules' and 'deficit of #7 causes certain condition'. After selection takes place, the tree is rotated so that the new root reflects the main topic amongst the preserved nodes, resulting in node 10 being root of (2b).
2. It is worth noting that Kintsch and van Dijk (1978) did not specify how many nodes can be recalled at a single time, however recent implementations (Fang, 2019) limit this number to at most 2.
At the beginning of cycle 3, the newly extracted nodes (14 -17) cannot be attached to the current tree because the linking node, #8, was prunned in the previous cycle. Therefore, information in proposition 8 is recalled and re-attached to the tree, showed as a squared node in tree (3a) and (3b). Then, the selection strategy is applied and the resulting tree is rebalanced, obtaining (3b).
After analyzing how trees are shaped in each cycle, it is important to point out their importance for the task of extractive summarization. Next, we elaborate on how memory trees can be leveraged for this end.

Properties Relevant to Summarization
The procedure for content manipulation described above imposes constraints on the shape, size, and content of memory trees during simulation. Such constraints bestow memory trees with special properties relevant to the task of summarization, specifically with respect to coherence, relevancy, and redundancy.
Local Coherence. A memory tree constitutes a connected structure in which two propositions are connected if any two of their arguments refer to the same concept. Connectivity, Kintsch and van Dijk (1978) argue, is a consequence of the text being well-structured and locally coherent, although connectivity is not a necessary condition for coherence -a disconnected structure can still be coherent for a reader. In this way, KvD enforces local coherence in a memory tree in the form of referential coherence. For instance, proposition 8 in cycle 3 of Figure 2 serves as a bridge to keep the memory tree connected, since propositions talking about cf patients (propositions 8 and 9) were discarded in the previous cycle.
This connectivity property has the following implication for local coherence in a final summary. By only updating the score of propositions in a memory tree, a summarization system can upgrade an entire set of coherent content units at the same time. As a consequence, coherent groups of propositions will be similarly scored, encouraging the selection of sentences that read more coherent as a whole.
Relevancy. In addition to being locally coherent, memory micro-structure takes the form of a tree for the following reason. KvD states that the root of a memory tree should contain information central to the argumentation represented in the working memory; hence, the root is deemed as the most relevant proposition in memory, and the more relevant a proposition is, the closer to the root it will be. This property could be exploited by a summarization system by designing a scoring function that takes the position of a tree node into account.
However, a KvD-based sentence ranking system that relies on proposition scoring would first need to capture the right propositions in working memory. Let us look at the first sentence of the gold summary in Figure 2). On the one hand, many propositions (7, 8, 12, 13, and 15) appear verbatim in this sentence, although sometimes only partially (e.g. 7 and 15). The capture of proposition 8 in cycle 3 highlights the importance of the recall mechanism in KvD to bring back relevant information. On the other hand, fine-grained information relevant to the summary might also be lost, such as node 14, in which a crucial property of a noun is not captured ('pulmonary').
Cycle 1 in healthy people , reactive oxidant species are controlled by a number of enzymatic and nonenzymatic antioxidants .
14: inflammation(pulmonary) 15: inflammation(in:#8) 16: contributes(#15,to:depletion) 17: of(depletion,antioxidants) Gold Summary patients with cystic fibrosis ( cf ) show decreased plasma concentrations of antioxidants due to malabsorption of lipid soluble vitamins and consumption by chronic pulmonary inflammation . carotene is a major source of retinol and therefore is of particular significance in cf . ... Figure 2: Simulation of KvD reading during three cycles. Each row shows the sentence consumed (top), the propositions extracted (left), and memory trees before (1a, 2a, 3a) and after (1b, 2b, 3b) applying a memory constraint of 5 nodes per cycle. Argument #N means that proposition N is used as argument. Squared nodes are recalled propositions. Solid lines connect nodes selected to keep in memory, and dotted lines connect nodes to be pruned or forgotten.
Redundancy. Finally, KvD processes influence redundancy reduction in two accounts. First, propositions in a memory tree are connected such that each proposition add new details about a concept without encoding more redundant arguments than necessary. For instance, consider again proposition 2 and 3 in Figure 2, where both propositions add relevant details (reactive and oxidant) about a concept (species). Hence, memory trees constitute a representation with the maximum amount of relevant details that can fit in working memory whilst minimizing the redundancy of arguments. Second, in case the recall mechanism needs to be used, KvD retrieves only the minimum amount of propositions to serve as bridge and connect the incoming propositions. Specifically, the recall mechanism only adds one recall path to the memory tree instead of many other alternative paths. By not loading redundant paths into memory, a system could avoid increasing the score of redundant content and update only one recall path at a time. This behavior, as we will demonstrate later, contributes immensely to decrease redundancy in the final summary and becomes particularly important for highly redundant documents, e.g. scientific articles that repeat information in several sections.

Summarization as Memory Simulation
The KvD theory provides a principled way to operationalize the manipulation of content units during reading and is precise in many aspects of the simulation, e.g. the nature and properties of memory trees. However, Kintsch and van Dijk (1978) make clear that the theory does not specify details of cognitive processes involving inference, i.e. the KvD theory can tell you when an inference occurs and its end result will look like but the theory cannot tell you how this end result is arrived at. Examples include how to construct propositions from text or how many nodes are retrieved by the recall mechanism. Hence, a computational implementation of this theory calls for design choices that allow us to define a tractable model with the NLP tools we have nowadays.
We posit the task of extractive summarization as the task of scoring the sentences in a document followed by a selection step in which an optimal set of sentences is chosen as the summary. In this section, two content scoring systems are introduced, TreeKvD and GraphKvD, which at their core simulate human working memory during reading, according to KvD theory. We start by elaborating on the procedure used to build propositions from syntactic structures automatically extracted from text. Then, we present our content scoring systems and elaborate on the design choices made during their implementation. Finally, we posit sentence selection as a knapsack optimization problem over a budget of tokens.

Proposition Building
Propositions are obtained by recursively merging and rearranging nodes in dependency trees, extending the procedure of Fang (2019). Given sentence s = w 0 , w 1 , ..., w N and its corresponding dependency tree Q with nodes {q 0 , .., q N }, 3 the objective is to obtain proposition tree P with nodes {p 0 , ..., p M }, M ≤ N , as follows.
First, we merge dependent nodes into head nodes in Q in a bottom-up fashion. Given u, v ∈ Q where u is head of v, operation merge(u, v) adds all tokens contained in v to node u and transplants children(v) -if any-to children(u). Let dep u,v be the grammatical relation between u and v, dependant v is merged into head u if and only if • Node u is a nominal or non-core dependant of a clausal predicate and v is a function word or a discourse modifier (e.g. interjections or non-adverbial discourse markers).
• Node u is any kind of dependant of a clausal predicate and v is a single-token modifier.
• Nodes u and v form part of a multi-word expression or a wrongly separated token (e.g. dep u,v = goeswith).
Consider the example in Figure 3. Starting from dependency tree Q (Fig. 3a), singletoken modifiers are collapsed into their head nodes (e.g. merge(model,this) and merge(galaxy,of)), and compound phrases are joint (e.g. merge(formation,galaxy)).
Second, we promote coordinating conjunctions to head status as follows. Given u, v ∈ Q, let v be a node with relation cc among children or grandchildren of u. We transplant node v to u's position and put u and all its children with relation conj as children of v. In our example ( Fig. 3.b), node 'and' is promoted and nodes 'galaxy formation' and 'the star burst' are transplanted as its children. Note that at this point in the procedure Q is still a tree (Fig. 3.c) but its nodes might now contain more than one token.
Then, for each non-leaf u ∈ Q we build proposition p = w u (arg v 0 , arg v 1 , ...), where w u is the sequence of tokens contained in node u and v i ∈ children(u). We set Fig. 3.d, and(galaxy formation,$4), presents proposition 4 as one of its arguments since node 'the start burst', from which proposition 4 is derived, is not a leaf.
Finally, edges between nodes in Q are used to connect their corresponding propositions and form proposition tree P , and we say that two propositions are connected if one proposition has among its arguments a pointer to the other proposition. For instance, proposition 1 in Fig. 3.d points to propositions 2 and 3 and hence, they are connected in P .
Under this procedure, connection among propositions in the same sentence takes a syntactic nature. However, propositions from different sentences -and hence different proposition trees-can still be connected if the lexical overlap amongst their arguments is strong enough. Next, we define connection through proposition overlap and how it is quantified.
Proposition Overlap We connect propositions from different sentences by quantifying the lexical overlap between their functors -predicates and arguments. Given p 1 ∈ P x and p 2 ∈ P y , , let A * (p 1 , p 2 ) be the optimal alignment between f unctors(p 1 ) and f unctors(p 2 ). Alignment A * is defined as the maximum matching that can be obtained greedily in the weighted bipartite graph formed from sets f unctors(p 1 ) and f unctors(p 2 ). The edge weight between two functors is defined as e(a, b) = jackard(L a , L b ), the Jackard similarity between their sets of lemmas after discarding stopwords, punctuation, and adjectives -L a and L b .
Then, the average overlap score between p 1 and p 2 , φ(p 1 , p 2 ), is defined as (1) This overlap score function becomes useful when searching an appropriate place to attach incoming propositions to the current memory tree or to pull propositions from long-term memory. We elaborate more on this in the next section. Step-by-step construction of proposition tree from an input sentence, starting from obtaining its dependency tree in UD format (a), merging dependent nodes into head nodes (b), promoting coordinating conjunctions to head status (c), to finally build propositions from non-leaf nodes (d).

TreeKvD
The first proposed system, TreeKvD, models how content is moved from working memory to long-term memory and vice versa. Working memory is represented as a proposition tree, pruned at the end of each cycle in order to simulate short-term memory limitations in humans. In contrast, long-term memory is represented as a forest of proposition trees, populated by subtrees demoted from working memory as reading progresses.

Memory Simulation
We now give an overview of the simulation procedure, outlined in Algorithm 1. The algorithm consumes a document D = s 0 , .., s k , ..., s |D| iteratively in memory cycles, updating working memory structure T and long-term memory structure F in each cycle. At the beginning of each cycle, the algorithm reads one sentence from input and extracts its proposition tree (line 5) following the procedure presented in section 4.1. Then, fresh propositions are added to the current memory tree (line 6) and the resulting tree is pruned to a constant size (line 9) in order to simulate human memory constraints. Pruned nodes and subtrees are then moved to the long-term memory pool. Finally, the score of the remaining nodes in memory tree are updated (line 10) and simulation continues to the next cycle. We now elaborate on the details of each step in the algorithm.
Attaching incoming nodes. Let s k be the sentence read in cycle k and P k = {p 0 , ..., p N } its corresponding proposition tree, and let T be the memory tree at the beginning of the cycle. We define the optimal place to attach P k to T as the pair (t, p) where t ∈ T, p ∈ P k such that (t, p) = argmax where φ(·) is the proposition overlap function defined in Equation 1. In case that no attachment point can be found, i.e. φ(t,p) = 0, ∀t ∈ T ∧ ∀p ∈ P k , algorithm 1 resorts to two cascaded backup plans. As first backup attachment plan, Algorithm 1 recalls forgotten propositions from longterm memory F to serve as bridge to connect P k and T . We define the optimal attachment place aided by node path where f −1 is the last node in path f and R is a parameter denoting the maximum allowed length of a recalled path.
In case that no suitable attachment can be found (total overlap score is still), procedure attachPropositions returns flag attached=False and the algorithm resorts to the second backup plan. This plan consists of deciding whether to keep T as memory tree during the current cycle or whether to replace it with P k . Memory tree T is replaced by P k if and only if |P k | > |T | and closeness(root(P )) > closeness(root(T )), where closeness(·) denotes the closeness centrality score of a node. 4 Under these conditions, a memory tree T could remain untouched in memory for many cycles, a phenomenon we name tree persistence. A highly persistent tree is undesirable since it can potentially block important connections between fresher propositions. In order to avoid this scenario, we reset the memory tree (line 16) if its persistence reaches the maximum permissible value, Ψ. Furthermore, we avoid over-scoring nodes in persistent trees by only updating their score if any form of attachment took place, line 10.
Choosing and adjusting the root. Let T be the proposition tree resulting from attaching P k to T . After the attachment takes place, we select the most appropriate node in T as the root, in line 8. An important property of working memory trees in the KvD theory is that the root conveys the most central topic at the time of reading. We model this property by selecting the node with highest closeness centrality score as the root. Such a root would facilitate reaching all nodes in the least amount of steps -in average-, a desired property during pruning.
Pruning working memory. Given a memory capacity of WM, the algorithm proceeds to select subtree T ⊂ T s.t. |T | ≤ WM in the following manner, line 9. Starting from the root, T is traversed in topological order until reaching a leave, adding each visited node to T . At this point, if |T | < WM, nodes are added in breath-first traversing order until |T | = WM or until all nodes are traversed. Finally, nodes and subtrees not included in T are moved to the long-term memory pool F and T is used as the new memory tree.

Proposition scoring
All propositions in document D start with a global score of zero. Every time a proposition is selected to stay in memory, i.e. ∀t ∈ T in cycle k, its global score is updated as follows where c(t, T ) quantifies the importance of proposition t on T by taking into account its position in T and how much information the subtree rooted at t holds. Formally, where depth(t) is the depth of node t with respect to the root and |T t | is the size of the subtree rooted in t.

Limitations
The presented system closely follows mechanisms of memory organization theorized by Kintsch and van Dijk (1978). As such, the system presents a number of processing limitations inherent to the KvD theory itself which we now elaborate on.
First, the constrained amount of content units in working memory at any given time poses a limitation to how much information the system has access to when updating the score of memory tree nodes. It is entirely possible that some propositions are pruned away and never recalled again, in which case its score will be zero.
Second, Kintsch and van Dijk (1978) define the recall mechanism as a routine capable of pulling an unlimited number of propositions from long-term memory. Additionally, propositions might not be recalled verbatim but simplified, given that the difficulty to recall specific details increases over time (Postman & Phillips, 1965). In system TreeKvD, we limit ourselves to recall previously read propositions verbatim and further limiting the maximum number of propositions to recall. This design choice limits the possibility of recalling important propositions back into working memory.
Third, attachment of an incoming proposition tree to the current memory tree is done by connecting one node in memory tree to one node in the incoming tree. Whilst this strategy guarantees that the resulting structure remains a tree, as KvD requires, many potentially useful connections are ignored. We address these limitations in the design of the next system.

GraphKvD
The second proposed system, GraphKvD, considers instead a single underlying structure for long-term memory and short-term memory. Long-term memory is modeled as a weighted undirected graph of propositions where edge weights represents strength of argument overlap. Working memory, instead, is modeled as a subgraph of long-term memory that preserves properties of KvD micro-structure, i.e. a tree with constrained size. Such modeling of memory modules allows for richer connections between incoming proposition trees and working memory, in addition to giving the system efficient access to nodes connected to memory tree nodes, significantly increasing the coverage of content during scoring. We now proceed to elaborate on how GraphKvD simulates human memory.

Memory Simulation
Memory processes at the micro level are operationalized by Algorithm 2, for which we now present an overview. Similarly to TreeKvD, processing is done in cycles, at the end of which working memory is represented as a tree of propositions with at most WM nodes. In each cycle, a proposition tree is obtained from an incoming sentence (line 6) and each node is connected to the current memory tree, potentially making use of a recall mechanism (line 7). Since the resulting memory structure is no longer a tree, the maximum spanning tree in this structure is selected as new memory tree (line 8). Then, this new tree is pruned to have at most WM nodes, which are then scored along with their neighbors, line 10. We now proceed to elaborate on the details of each step.
Attaching incoming nodes. Given s k and P k , the sentence read in cycle k and its corresponding proposition tree, and current memory tree T , we define undirected, weighted . Let G P and G T be the subgraph composed of nodes and edges imported from P k and T , respectively. Then, node (t, p). The resulting working memory structure, G , will then be used to update the long-term memory graph, G, and build a new memory tree for the next steps in the current processing cycle.
Note that as per KvD requirement, a representation of working memory must be coherent, i.e. G must be a connected graph. This is not true when no attachment was made, i.e. all overlap scores were zero. In this scenario, Algorithm 2 resorts to backup attachment plans similar to TreeKvD.
As first backup plan, Algorithm 2 employs a recall mechanism in order to retrieve paths from G that would connect each node in P k to each node in T . Given node p ∈ V [G P ], the optimal attachment place t ∈ G T aided by path wheref is the shortest path from f 0 to t with length at most R, obtained from an unweighted version of G. Note that when G is a disconnected graph, path f might not exist and hence, connection between G P and G T will not be possible. In this case, Algorithm 2 resorts to a second backup plan, as follows. Similarly to TreeKvD, G T is discarded from G if |P k | > |T | and closeness(root(P k )) > closeness(root(T )), or if the tree persistence has reached its allowed limit, ψ = Ψ, line 17.
In case G T is discarded, however, we proceed to enrich G with nodes retrieved from G. For each node p ∈ V [G ], we retrieve candidate nodes in the following order. Nodes from the current context sentence s k was read, i.e. nodes belonging to the current paragraph or section, are retrieved first. Then, nodes are retrieved in inverse order of processing recency, i.e. propositions from sentences processed at the beginning of the simulation are retrieved first. This particular retrieval order follows free recall accuracy in human subjects (Glanzer, 1972). 5 In particular, recall of early processed information is significantly more accurate and depends on long-term memory. Recall of recent items is also very accurate, a tendency called recency effect, and it depends of short-term memory.
After candidate enrichment nodes are retrieved, each node in G is connected to its top K candidates, sorted by argument overlap score. Note that during this operation, only G was expanded and G remained untouched. Next, we elaborate on how long-term memory graph G is updated.
5. Free recall is a technique used in psycholingusitic studies of human memory in which a subject is presented with a string of items and is free to recall them in any order; in contrast, serial recall requires the subject to recall the items in order.
Updating memory structures. After attaching incoming proposition nodes to memory tree T , the resulting structure, G , is added to long-term memory graph G, line 8. Note that G might not constitute a tree and hence, not be a valid KvD micro-structure of memory. Therefore, we define tree T as the maximum spanning tree in G and choose as root the node with maximum closeness score, line 9. Then, T is pruned down to have at most WM nodes using the same strategy as in section 4.2, line 10. Finally, nodes in the resulting tree, T , are scored and their global scores updated. Next, we elaborate in how GraphKvD scores nodes in memory trees.

Proposition Scoring
Given memory tree T with size |T | ≤ WM during cycle k, we define where c(x) is scoring function defined in section 4.2.2, and γ < 1 is a decay factor. In this way, propositions that contribute to the understanding of nodes in T are reinforced, and the more a proposition is selected the more its connections are updated.

Simulation Example
Next, we illustrate the procedures outlined in Algorithms 1 and 2 with an example, showcased in Figure 4. The example takes two sentences from a scientific article and simulates two memory cycles with TreeKvD (left) and GraphKvD (right). The propositions involved (middle row) in the cycles are presented alongside the corresponding gold summary (bottom row). Propositions not directly mentioned in the simulation but necessary for content interpretation are showed in italic. First, we analyse the processes involved during attachment in a memory cycle, including how recall mechanism operates. Then, we relate the properties a memory tree should exhibit according to the KvD theory, and the properties of memory trees obtained with TreeKvD and GraphKvD.
Memory Cycles. In cycle k, both systems manage to attach the incoming proposition tree P directly to the current memory tree T k−1 , with such connections illustrated as red dotted lines in Figure 4. Notice that TreeKvD is allowed to make only one connection (79 → 81) so that the resulting structure, T , remains a tree. In contrast, GraphKvD is allowed to connect each node in P back to T (e.g. 84 → 79, 85 → 71), which results in structure G , an undirected weighted graph. After choosing the new root (node 81), the retention process (function memorySelect) selects the new memory tree T k .
In the next cycle, k + 1, the incoming P cannot be attached directly to T k and hence, the recalled mechanism is used. TreeKvD recalls a 3-node path to connect node 88 to 81, linking information about proposed models ('models for turbulence' in 81) to methodology ('scaling methods' in 25, 24, 21) and hypothesis exploration ('we try to see if these suggest' in 88). In contrast, GraphKvD recalls a single node linking the studied phenomenon ('mhd turbulence' in 81) to its properties of interest ('such relations' in 79, making reference to information in 75) and to the specific property being studied ('bridge relations' in 90).
Properties of Memory Trees. Properties of memory structures at the micro level, as discussed in Section 3.2, have the potential to greatly influence the level of local coherence and redundancy in output summaries, in addition to identifying relevant content to be included. We now elaborate on how this influence manifests in our example.
First, regarding local coherence, a connected memory tree is evidence that content units currently held in memory are not a disjoint set of mutually exclusive concepts but a set that can be interpreted in a coherent manner. For instance, the content in T k−1 could be verbalized in the following manner: We examine dynamic multiscaling...in a shell model for 3d mhd [71,72] and scalar turbulence [80]. Dynamic multiscaling exponents are related by linear bridge relations to equal time multiscaling exponents [75]. We have not been able to find such relations for mhd turbulence so far [77,78,79].
where the propositions used to verbalize each phrase or sentence are indicated inside square brackets. As can be seen, the text above reads smoothly and exhibits an acceptable level of lexical cohesiveness and co-referential coherence. By updating the score of a set of propositions capable of forming a coherent text, a KvD system encourages the similar ranking of mutually coherent propositions. Hence, a content selector is also encouraged to select a set of sentences exhibiting a non-trivial level of lexical cohesiveness.
A similar reasoning can be applied to explain the influence of memory simulation over redundancy in output summaries. As claimed in Section 3.2, a memory tree constitutes a non-redundant set of propositions, with each proposition adding details of an entity or topic shared with the propositions it is connected to. For instance, node 81 adds information about 'mhd turbulence' to T k−1 when connected to node 79. Moreover, when the recall mechanism is used in cycle k +1, only one recall path is added to T k (25, 24, 21 in TreeKvD and 79 in GraphKvD) instead of many potentially redundant recall paths. Hence, by updating the score of a minimally redundant set of propositions in each cycle, a KvD system encourages non-redundant content to be ranked closely and by extension, the content selector is encouraged to select sentences with an acceptable level of redundancy.
Finally, memory trees are capable of identifying and ranking relevant propositions, hence encouraging a selector to pick sentences with relevant content. In our example, we observe that both TreeKvD and GraphKvD retain propositions 81, 84, 85, 86 in T k and T k+1 . These propositions cover information directly mentioned in the gold summary, coloured in blue in Figure 4.

Content Selection
Previously, we elaborated on how to discretize content units as propositions and introduced symbolic systems that assign a score to each one. In this section, we elaborate on how to select sentences from the source document based on its the scored propositions.
Previous work has pointed out that automatic summarization metrics based on n-gram overlap, such as ROUGE score, are sensitive to number of tokens in produced summaries, and summarization models should only be compared against each other if they produce summaries of similar length (Narayan et al., 2018a;Schumann et al., 2020). Given that reference summaries in recent summarization benchmarks (Hermann et al., 2015;Grusky et al., 2018;Cohan et al., 2018) present a high variance in terms of number of tokens, recent extractive summarization approaches follow the common practice of extracting a fixed number of sentences regardless of their length. However, this extractive setup makes comparison between systems highly unreliable w.r.t. automatic metrics, specially if length statistics of produced summaries are not reported. In contrast, summarization setups at Document Understanding (DUC) and Text Analysis conferences (TAC) constrained extracted summaries to a fixed number of tokens (Harman & Over, 2004;Over et al., 2007). Once more, such a harsh constraint can lead to unreliable system comparison on benchmarks that present reference summaries with high variance in summary length.
For this reason, we take a middle ground approach to control for summary length, more strict than a fixed number of sentences and more permissive than a fixed number of tokens. We introduce a sentence selector that is allowed to build summaries within a range length in number of tokens. In this way, the variability of automatic metrics due to extreme differences of number of tokens in compared summaries is explicitly mitigated.
Formally, the problem of selecting a subset of sentences from a document is defined as a 0-1 knapsack problem with a soft constraint over number of tokens. The objective is to maximize the total score of selected sentences while complying with a soft budget of W ± σ tokens. Each sentence s i ∈ D contributes to the budget with its length in number of tokens and score sc(s i ) = p∈P (s i ) score(p), the sum of scores over all propositions extracted from : suggest(if, these, any bridge relations) Gold Summary we present the first study of the multiscaling of time dependent velocity and magnetic field structure functions in homogeneous , isotropic magnetohydrodynamic ( mhd ) turbulence in three dimensions . we generalize the formalism that has been developed for analogous studies of time dependent structure functions in fluid turbulence to mhd . by carrying out detailed numerical studies of such time dependent structure functions in a shell model for three dimensional mhd turbulence , we obtain both equal time and dynamic scaling exponents . sentence s i . Hence, candidate summary S ⊂ D with N (S) tokens, is defined as where | · | denotes absolute value. We denote this selector as Knapsack. Note that our Knapsack selector can be plugged into any summarization system that produces a score for each sentence.

Model Parameters
We investigate the performance of both our systems, TreeKvD and GraphKvD, across a number of values of working memory size, WM= {5, 20, 50, 100}, and set the maximum recall path length R = 5 and maximum tree persistence Ψ = 5. For TreeKvD, search for the optimal recalled path is finished early if the total overlap score of a candidate path is higher than 0.5. For GraphKvD, we use a decay factor γ = 0.01.

Comparison Systems
We report results on a range of standard baselines as well as unsupervised systems. Standard extractive baselines include Lead (selecting the leading sentences of a document until budget is met), Random (randomly sampling sentences following a uniform distribution), and Random-Wgt (sentences are sampled with probability proportional to the length of the section they belong to). Regarding unsupervised systems, we report results on TextRank (Mihalcea & Tarau, 2004) 6 and PacSum (Zheng & Lapata, 2019), which models sentences as nodes in a graph, learns a specialized edge scorer, and scores sentences based on node centrality. For computational purposes, we limit connection to sentences in a window of size 50. 7 We employ SciBERT (Beltagy, Lo, & Cohan, 2019) as base edge scorer and report results on two configurations. The first, labeled simply as PacSum uses default hyperparameters reported by Zheng and Lapata (2019). The second, PacSum-FT, is finetuned over a sample of 1000 documents following the procedure therein. Furthermore, in order to evaluate the appropriateness of the iterative KvD content scorer in both our proposed systems, we compare against a proposition graph baseline in which the edge weight corresponds to argument overlap, labeled as FullGraph. Then, proposition scores are obtained by running PageRank (Brin & Page, 1998) over this graph. Similarly to PacSum, proposition connection is limited to those between a window of 50 sentences.
In addition, a supervised baseline is presented in order to present an upperbound on performance and better grasp an idea on the difficulty of the task. We extend SciBERT 8 by adding a linear classifier layer on top of the transformer model and fine-tune it over the same subset as PacSum-FT. In a similar fashion to Cohan et al. (2019), we consume each document in chunks of 5 sentences, train it for 2 epochs using a batch size of 8, optimize with Adam (Loshchilov & Hutter, 2018) with a fixed weight decay parameter of 0.1. Additionally, we use a slanted triangular learning rate scheduling with 10% of total 6. We use Gensim implementation (Rehurek & Sojka, 2010). 7. Such a limitation was possibly not considered by Zheng and Lapata (2019) since their model was not designed for long documents, it was tested on the CNN/DM dataset in which documents are 50 sentences long in average. 8. We use the pretrained model served by HuggingFace at https://huggingface.co/allenai/scibert scivocab uncased training steps as warm up and a top value of 1e −5 . Gradients are accumulated for 16 training steps and clip gradients by norm value at 0.1. We refer to this baseline as SciBERT.
Finally, we compared our proposed models against a previous implementation of the KvD theory (Fang, 2019), labeled as KvD-Fang. Note that all the aforementioned systems were modified to use our knapsack sentence selector in order to ensure a fair comparison between systems. The same soft-budget selection is applied to extract oracle summaries with the difference that the score of a candidate summary is the sum of ROUGE-1 and ROUGE-2 recall values -calculated with respect to the gold summary-instead of the sum of scores over all sentences.

Datasets
We used PubMed and ArXiv datasets collected by Cohan et al. (2018), composed of scientific articles in English. For each article, the source document is composed of all sections and the abstract is used as reference summary. We further preprocessed both datasets after noticing substantial sentence tokenization error and pollution of latex code. 9 For candidate summaries, a soft budget of 190 ± 50 and 205 ± 50 were used for ArXiv and PubMed, respectively, after a preliminary analysis of training data statistics.
For funetuning of baselines such as PacSum-FT and SciBERT, we randomly select 1 000 instances from the training set of each dataset under a uniform distribution.

Evaluation
Summary quality was evaluated automatically using F 1 ROUGE (Lin, 2004) which measures lexical -n-gram-overlap between extracted summaries and reference summaries, serving as an indicator for relevancy and fluency. Even though many issues have been identified when using ROUGE outside its proposed setting (Liu & Liu, 2008;Cohan & Goharian, 2016), many variations of the original metric have shown strong correlation with human assessment (Graham, 2015;ShafieiBavani et al., 2018;Fabbri et al., 2021). Nevertheless, ROUGE is not designed to appropriately reward semantic and syntactic variation in extracted summaries. For this reason, we also evaluate summaries using F 1 BertScore (Zhang, Kishore, Wu, Weinberger, & Artzi, 2019) which addresses semantic similarity by comparing contextual embeddings given by a pretrained BERT model. In all our experiments, we use SciBERT ) as core pretrained model and apply importance weighting to diminish the effect of non-relevant words (e.g. function words) on BertScore values. 10 We refer to this metric as SciBERT-Score, SciBS.
In terms of redundancy and local coherence, we employ a range of metrics which we now elaborate on.

Measuring Redundancy
Following previously proposed methodologies to measure redundancy in texts (Xiao & Carenini, 2020;Bommasani & Cardie, 2020), we report the following metrics over source 9. See Appendix A for details 10. IDF statistics were obtained from documents in the training set of each dataset. documents and summaries, each metric value is in the range of [0; 1] and the higher a metric value is the more redundant a text will be.
Uniqueness (Uniq). Adapted from Peyrard, Botschen, and Gurevych (2017), this metric defines n-gram redundancy in a text as U niq = 1−V (n-gram)/N (n-gram), where V (n-gram) is the number of unique n-grams and N (n-gram) is the number of total n-grams. We report the geometric mean among uniqueness values for unigrams, bigrams, and trigrams.
Sentence-wise ROUGE (RedRL). Adapted from Bommasani and Cardie (2020), this metric defines redundancy as the average ROUGE-L F1 score among all pairs of sentences. Given candidate summary S, RedRL = mean (x,y)∈S×S,x =y ROUGE-L(x, y).
In this work, the length of candidate summaries is controlled for and hence, pair-wise ROUGE scores among sentences in summaries presents an acceptable variance. However, this is not the case for input documents: the longer a document is, the longer the tail will be in the distribution of sentence-pair ROUGE scores; hence, a simple average over sentencepair scores would be skewed by the long tail. In order to mitigate the effect of document length on RedRL redundancy, we define RedRL D = mean x∈D max y inD,x =y ROUGE-L(x, y).

Measuring Local Coherence
A coherent and fluent text, longer than a single sentence, exhibits a non-trivial level of redundancy as a consequence of its lexical cohesiveness. For instance, observe the first two sentences in the example presented in Figure 1. Notice that the underlined phrases (wolf rayet and stellar winds) repeat in order to maintain cohesiveness between consecutive sentences. These two sentences would exhibit a redundancy value greater than zero according to the aforementioned metrics. However, this value would indicate lexical cohesiveness instead of content redundancy. It is then necessary to consider both lexical cohesiveness and redundancy metrics when assessing the quality of a candidate summary.
In this work, we measure cohesiveness of a text with its probability given by a language model. Formally, perplexity (PPL) of a candidate summary S is given by where P (w i |w −i ) is the conditional probability of token w i given by a pretrained language model, in our case, SciBERT .

Results and Discussion
In this section, we present results for our proposed systems, TreeKvD and GraphKvD, and comparison systems on the ArXiv and PubMed dataset. We discuss about the effectiveness of the proposed models on dealing with content redundancy and local coherence, while maintaining an acceptable level of relevancy. Then, quantitative and qualitative experiments provide evidence on how properties of simulated cognitive processes affect final summaries. Finally, we discuss the impact of the proposed content selector and its effectiveness at controlling for summary length.

Relevancy, Redundancy, and Local Coherence
We start by analysing the performance of our models in terms of relevancy, redundancy, and local coherence. Results on relevancy are summarized on Table 1 which presents ROUGE F1 scores and SciBERT-Score (SciBS), along with each system's summary length statistics (mean and standard deviation). First, it can be observed that the organization of information in scientific articles poses a challenge for trivial baselines, as evidenced by the low performance of Lead, Random, and Random-Wgt. Note that including document structure information helps Random-Wgt obtain better results than Random. Second, the sizable gap between the supervised baseline, SciBERT, and the other systems tells us just how hard the task is and what an upper bound in ROUGE values looks like. Note that finetuning over in-domain data gives huge improvements, as depicted by PacSum and PacSum-FT. When comparing unsupervised baselines, we find that TextRank is the best performing system w.r.t. relevancy scores (R1, R2, and SciBS) in both datasets, widening the gap in PubMed. Since both FullGraph and TextRank use PageRank to rank content, we can conclude that TF-IDF similarity between sentences is more robust than argument overlap similarity between propositions, as done by FullGraph. Moreover, it is interesting to note that TextRank even outperforms PacSum-FT, a baseline fine-tuned on the validation set, on both datasets. We hypothesize of two reasons behind this result. First, PacSum's finetuning procedure encourages strong connections between neighboring sentences. Hence, relying only on local coherence for scoring content might not be enough to capture relevant content. Second, PacSum was designed and tested on news articles which are significantly shorter than scientific articles, with relevant information more evenly distributed across the whole document. This, added to the vastness of knowledge in the scientific domain, could contribute to the lower-than-expected performance of PacSum systems.
When comparing among KvD systems in terms of relevancy scores, we observe that under the best configuration of working memory size (WM = 100), GraphKvD and TreeKvD systems perform in par, if not better, than unsupervised baselines including a previous implementation of KvD theory, FangKvD. Nevertheless, the lowest-performing KvD systems do not fall far behind, e.g. 1.92 ROUGE-1 points of difference for GraphKvD over the ArXiv test set. It is worth noting that performance increases as working memory capacity (WM) increases, outperforming FullGraph in both datasets and comparably to TextRank in ArXiv. However, the effect of WM is somewhat inverted in FangKvD, with performance decreasing as WM increases possibly due its reliance on external domain-dependant resources like WordNet.
Redundancy and Coherence in Summaries. Results of redundancy and local coherence metrics are summarized in Table 2 for both ArXiv and PubMed test sets. A higher redundancy score corresponds to a more redundant candidate summary, and the lower perplexity is the more locally coherent a summary potentially is. Notably, we also report redundancy scores and perplexity for reference summaries (Reference, last row) in order to have a reference point for what a desirable level of redundancy and coherence looks like. However, it is important to state that lower redundancy and perplexity scores do not necessarily correspond to higher relevancy scores. For instance, Lead obtains ROUGE scores lower than other comparison systems whilst obtaining low U niq, RedRL, and P P L  scores. In contrast, random baselines obtain significantly higher levels of redundancy and perplextity. This indicates that the relationship between relevancy and redundancy is a trade-off and systems should be encouraged to seek a middle ground. Let us first compare redundancy and coherence levels of gold summaries, Gold Summary in Table 2, against systems that leverage supervision signals: the extractive oracle (Oracle), SciBERT,and PacSum-FT. On the one hand, we find that Oracle is closer to gold summaries in terms of redundancy that any other system. In terms of uniqueness redundancy (U niq), Oracle is always below Gold Summary redundancy; however, in terms of rouge redundancy (RedRL), Oracle can be below (ArXiv) or above (PubMed). This result demonstrates the effectiveness of using ROUGE as guiding metric to extract oracle summaries that not only are relevant but also show an acceptable level of redundancy. On the other hand, SciBERT places farther away from Gold Summary than Oracle in terms of redundancy, although placing closer in terms of local coherence (P P L). This could evidence that the high redundancy levels are due to the training objective and not the training signal, given that SciBERT was trained using Oracle as reference. A simple sequence labeling objective, although effective at identifying highly relevant information, favours the selection of semantically similar sentences -hence more redundant-that nevertheless read cohesive. In contrast, the inter-sentential vicinity objective leveraged by PacSum-FT successfully exploits lexical cohesiveness among consecutive sentences, as can be seen from its low levels of redundancy and perplexity. However, this objective is extremely dependent  Table 2: Redundancy and local coherence levels (PPL) of candidate summaries at the system level over the test set of ArXiv and PubMed, alongside summary length in tokens (N (S)) reported in mean(standard deviation) format. (*) These systems leverage supervision signal, hence they are not strictly unsupervised.
on domain knowledge, as evidenced by the sizable increase in redundancy and perplexity scores when switching from a finetuning setup (PacSum-FT) to a completely unsupervised setup, i.e. without gold data to finetune the model, PacSum. When comparing completely unsupervised systems, we note first that TextRank and PacSum are placed among the highest in terms of U niq, RedRL, and P P L, meaning that they produce significantly more redundant and less locally coherent summaries that other systems. For TreeKvD and GraphKvD systems, we observe that higher working memory size leads to lower redundancy and higher local coherence. In general, TreeKvD is capable of obtaining lower redundancy levels than GraphKvD under the same working memory configuration whilst obtaining similar levels of perplexity. Instead, FangKvD systems exhibit an inverse redundancy trend: redundancy scores increase as working memory size increases, with the least redundant FangKvD system (WM = 5) still being more redundant than the least redundant TreeKvD and GraphKvD systems (WM = 100). These results demonstrate that the proposed KvD systems are capable of producing relevant summaries with much lesser redundant information without compromising local coherence.
Finally, note that the performance of FullGraph falls in-between KvD systems, obtaining lower scores than the most redundant KvD systems (WM = 5) but higher scores than the least redundant KvD systems (WM = 100). Given that FullGraph implements the same graph structure than GraphKvD but does not simulate memory (it has access to the complete graph at all times), these results indicate the importance of simulating memory of limited capacity.
Effect of Document Redundancy. Next, we take a closer look at the redundancy and coherence levels in summaries extracted from increasingly redundant documents. Figure 5 shows performance of summarization systems in relevancy (ROUGE-L), redundancy (RedRL), and local coherence (P P L, right column) across different levels of document redundancy (RedRL D ). The test sets were divided in bins according to their document redundancy score and the average metric value per bin is reported. For simplicity, we only plot performance of KvD systems with WM = 50 since other memory size configurations showed similar trends.
In terms of relevancy (left column), TreeKvD and GraphKvD show non-decreasing performance as document redundancy increases in ArXiv (top left), performing comparably to and even outperforming TextRank on highest levels of redundancy. In PubMed (bottom left), document redundancy is less impactful on KvD systems, performing comparably to TextRank on highest levels of redundancy.
In terms of summary redundancy (middle column), KvD systems exhibit significantly lower redundancy levels than other systems with the exception of Gold and Oracle. Note that TextRank and SciBERT are greatly impacted by document redundancy levels in both datasets. In ArXiv (top middle), redundancy in KvD systems decreases as document redundancy increases, a trend also showed by PacSum-FT. Instead, in PubMed (bottom middle), the impact of document redundancy is less significant on KvD systems, which remain the least redundant systems after Oracle. It is worth noting that PacSum-FT shows robustness to document redundancy, albeit with a significantly higher redundancy level that KvD systems.
In terms of local coherence in summaries, as measured by perplexity (PPL), it is desirable that the coherence level in candidate summaries is close to that of the reference summaries, Gold, and consequently close to Oracle. In both datasets, SciBERT and PacSum-FT remain close to Gold followed by KvD systems. For highly redundant documents in PubMed, KvD systems report lower levels of perplexity than PacSum-FT, comparable to Oracle. However, it is worth noticing that SciBERT and TextRank eventually report lower levels of perplexity that Gold, at high levels of document redundancy in both datasets. We hypothesize that these systems select increasingly fewer sentences, eventually selecting only one -very long-sentence as summary, hence the low perplexity.
Extraction across a document. We investigate from which part of a document our proposed systems extract sentences. From Figure 6, we observe that system subject to supervision, SciBERT and PacSum-FT, tend to extract sentences from the very beginning of a document. In contrast, TreeKvD and GraphKvD select sentences from all over the document in similar fashion to Oracle.
Qualitative Analysis. We provide a qualitative analysis of system summaries extracted from a highly redundant document (RedRL D = 63.34%) in the ArXiv dataset, showcased in Figure 7. Each system summary is accompanied by its relevancy (ROUGE-L, RL), redundancy (RedRL) and local coherence score (P P L), as well as its length in number of tokens, N (S). First, it can be observed that all system outputs are of comparable length (between 189 and 196 tokens) and of comparable level of relevancy, i.e. similar RL score, with the exception of Oracle. Second, we observe that a redundancy level lower than the reference (Oracle and Gold Summary) implies a level of local coherence (higher P P L) lower than the reference, as seen in TreeKvD[50] and GraphKvD [50]. However, redundancy levels higher than the reference can lead to -apparently-higher levels of local coherence (lower P P L), as seen in TextRank. The centrality-based scoring implemented in TextRank seems to favour the selection of sentences with as many tokens in common as possible, e.g. 'monopole', 'quadrupole', 'mode', and 'frequency', coloured in red. Most critically, TextRank is susceptible to select repeated sentences 11 such as sentences 68 and 87 (position index is showed in squared brackets).

How Simulated Cognitive Processes Affect Final Summaries
The KvD theory describes cognitive processes involved in short-term memory manipulation and constraints over memory structures. While it is well understood how these processes and constraints would influence reading comprehension in a simulated environment, it is less intuitive to establish how they influence content selection during summarization. In this section, we shed light on how final summaries are affected by working memory capacity and the shape of memory trees.
Working Memory Capacity (WM). Intuitively, the more memory capacity a KvD system has, the more propositions it will be able to retain in memory, increasing the chances that relevant propositions are scored higher and are eventually selected for the final summary. This is evidenced by the consistently increasing ROUGE and SciBERT-Score values for increasing memory capacity, WM, as showed in Table 1. However, KvD systems with WM = 100 obtain comparable or higher scores than FullGraph, a system that does not simulate working memory and which scoring strategy has access to all the propositions in a document at all times. This indicates that constraining the size of the memory tree in each iteration encourages KvD systems to retain propositions that are crucial for comprehension of the latest coming information. Without a constrained working memory, the retention step (function memorySelect in Algorithms 1 and 2) is exposed to content irrelevant to the current local context. Similar conclusions can be drawn with respect to summary redundancy: more working memory capacity leads -rather counter-intuitively-to less redundant summaries. We hypothesize that summary redundancy is affected in the following way. When a new sentence is read, a KvD simulator attaches its proposition tree to the current memory tree, as elaborated in section 4. A greater memory capacity would mean that more propositions from the new sentence could remain in memory if selected. In a reasonably cohesive text, this leads neighbouring sentences to obtain similarly high scores. As a consequence, the selector is encouraged to extract sentences close to each other since they show similar scores; hence, obtaining a more locally coherent summary. For instance, in Figure 7, TreeKvD selects sentences 24 and 25, whilst GraphKvD selects 112 and 113.
Another aspect greatly influenced by working memory capacity is that of how much information in the source document can be covered. As noted in Section 4.3, it is possible that some propositions are pruned away and never recalled again, in which case their final score will be zero. We say that a proposition is covered by a KvD system if such proposition appears at least once in a pruned memory tree during simulation. Furthermore, we define document coverage as the ratio of covered propositions over the total number of propositions in a document. Not surprisingly, we found that increasing working memory capacity increased document coverage in both TreeKvD and GraphKvD. When WM = 5, TreeKvD is able to cover 62% of all document propositions in the ArXiv test set, and up to 96% when WM = 100. GraphKvD further improves coverage to 78% at WM = 5 and 97% at WM = 100. However, notice that FangKvD exhibits a much lower coverage: 22% when WM = 5 and up to 44% when WM = 100. We hypothesize that the drastic improvement in GraphKvD is due to the diffusion mechanism which updates scores of direct neighbours of memory tree nodes. Similar trends were observed in the PubMed dataset. These results lay down evidence that the proposed computational implementations of KvD theory are effective at covering most -if not all-content units in a document during simulation.
So far in our analysis we have considered memory capacity as a hyper-parameter of a KvD system, expected to remain fixed throughout the entire simulation and fixed for all documents in an evaluation set. The following question then arises when looking at each sample individually: what is the right capacity of working memory in order to produce a summary with the most relevant content? We attempt to answer this question by selecting for each sample in the validation set, the working memory size WM that yields the highest sum of ROUGE-1 and ROUGE-2 scores. The results are encouraging: when using the best possible WM per sample in ArXiv, TreeKvD exhibits an increase in absolute points of 3.19 in ROUGE-1, 2.36 in ROUGE-2, and 2.86 in ROUGE-L. This is compared to the best performing configuration, i.e. when using WM = 100 for all samples, TreeKvD[100]. Most surprisingly, the distribution of best WM per sample is rather balanced, with 26.5% of samples preferring a WM = 100, 26.7% a WM = 50, 24.11% a WM = 20, and 22.5% a WM = 5. GraphKvD exhibits a similar increase of 3.06, 2.33, 2.75 in ROUGE-1, ROUGE-2, and ROUGE-L, respectively. A similar trend was observed on the validation set of PubMed. However, it should be noted that we did not find any strong correlation between working memory capacity and ROUGE or SciBS scores, which indicates that the ability of a KvD system to produce relevant summaries is not influenced by its working memory capacity. Instead, we suspect that memory capacity might be an indicator of text difficulty or cognitive easiness, however the exploration of this hypothesis falls out of the scope of this work and we leave it to future investigations.
Working Memory as a Tree. Next, we investigate the impact of leveraging the position of a node in the memory tree structure during scoring, in terms of relevancy and redun-dancy scores. We compare the scoring function in Equation 4 against two other strategies. The first one, denoted Freq, consists of a frequency heuristic, c(t, T ) = 1, ∀t ∈ T , which only counts how many memory cycles a proposition participates in. The second strategy, denoted Eigen, scores nodes based on their eigen-vector centrality, i.e. c(t, T ) = where λ is the largest eigen-value of the adjacency matrix of T . 12 Figure 8 shows the performance of TreeKvD (right) and GraphKvD (left) over the ArXiv (top row) and PubMed (bottom row) validation sets. Systems using scoring function c(t, T ) in Eq. 4 are labeled with Tree, e.g. TreeKvD [Tree]. First, we observe that Tree scoring significantly outperforms Eigen and Freq scoring, for all values of working memory capacity. This results demonstrates the advantage of modeling memory as a tree structure and leveraging the position of a node for scoring, compared to just considering memory as a bag of content units (as Freq does) or even using complicated centrality strategies, as done by Eigen. However, it is worth noticing that for GraphKvD, the gap between Tree and Eigen diminishes as WM increases, even performing comparably in ArXiv. This might indicate that GraphKvD is superior than TreeKvD at placing highly influential (i.e. relevant) nodes closer to the root, in which case the proposition ranking given by Tree and Eigen is highly similar.

Controlling for Summary Length
Finally, we analyze the effect of the proposed Knapsack selector on summary length. From Table 2, it can be observed that all systems present length distributions centered closely at the predefined budget (190 for ArXiv and 205 for PubMed) with very low variance, with the exception of Lead and Oracle. These two systems present high variance in summary length due to their special selectors. Oracle uses a knapsack selector guided by ROUGE scores capable of stopping selection early if the ROUGE score starts to decrease, potentially leading to summaries shorter than the budget. Instead, Lead is susceptible to the length of the first sentences in the document, leading to summaries longer than the budget for both analysed datasets.
Then, we compare the Knapsack selector against a greedy selector, i.e. a selector that extracts the highest-scored sentences one by one until budget is met (or surpassed).

Conclusions
In this paper, we studied the trade-off between redundancy and local coherence in summaries produced by unsupervised extractive systems when the input is a long document that exhibits information redundancy among the parts it is divided in. As case study, we experimented with scientific articles for which the main body -divided in sections-is considered as the input document and the abstract is considered as the reference summary. Two summarization systems were introduced, capable of closely simulating how information is discretized and organized in human working memory in a complete unsupervised way, according to the Micro-Macro Structure theory of reading comprehension. On the one hand, results from automatic evaluation show that our systems produce summaries with a consistent level of redundancy across varying levels of input document redundancy, whilst maintaining levels of local coherence comparable to those in reference summaries. Instead, the compared systems produce increasingly redundant summaries as the document redundancy level increases, confirming insight from previous work on redundancy (Xiao & Carenini, 2020).
On the other hand, extensive experiments lay evidence that mechanisms operating in the simulated working memory (e.g. recall of specific content from long-term memory, or how to select content that remains in short-term memory), as well as properties of the derived memory structure (e.g. working memory size or position of content units in the structure) significantly improve performance in terms of automatic metrics for relevancy, when compared against systems that do not implement these features.
Finally, we proposed a content selector capable of controlling for summary length in terms of number of tokens, ensuring a fairer comparison among summarization systems. Experiments show that systems employing the proposed selector are able to produce summaries much closer to the budget than greedy selectors, resulting in a summary length distribution centered at the budget and with low variance. we study the collective excitations of a neutral atomic bose einstein condensate with gravity like interatomic attraction induced by electromagnetic wave . using the time dependent variational approach , we derive an analytical spectrum for monopole and quadrupole mode frequencies of a gravity like self bound bose condensed state at zero temperature . we also analyze the excitation frequencies of the thomas fermi gravity ( tf g ) and gravity ( g ) regimes . our result agrees excellently with that of giovanazzi et al , which is obtained within the sum rule approach . we also consider the vortex state . we estimate the superfluid coherence length and the critical angular frequencies to create a vortex around the -axis . we find that the tf g regime can exhibit the superfluid properties more prominently than the g regime . we find that the monopole mode frequency of the condensate decreases due to the presence of a vortex .
Oracle 68.52 40.30 3.13 195 [13] the purpose of this paper is to give an analytic expressions for collective excitation frequencies , superfluid coherence length and critical angular frequencies required to create a vortex of a rotating bose condensed state and to compare qualitatively the results of the tf g regime with the results obtained in the tf regime of an ordinary atomic bec . here , by using the time dependent variational method , we obtain the excitation spectrum of a gravity like self bound bose gas . [14] we also calculate the lower bound of the ground state energy , monopole and quadrupole mode frequencies of the tf g and the g regimes .
[15] our result agrees excellently with that of ref . which is obtained within the sum rule approach .
[17] we estimate the superfluid coherence length and the critical angular frequencies required to create a vortex along the -axis .
[19] we find that the monopole mode frequency of the condensate decreases due to presence of the vortex .
[159] we could say that the tf g regime should exhibit the superfluid properties more prominently than the g regime .

TreeKvD[50]
39.87 39.29 3.54 189 [7] in this system , the gravity like attraction balances the pressure due to the zero point kinetic energy and the short range interaction potential .
[24] the bec of charged bosons confined in an ion trap can be described by the above mentioned lagrangian if we set @xmath , where @xmath is the electronic charge . to calculate the excitations spectrum of an atomic bec with gravity like interaction , we will use the time dependent variational method .
[25] this technique has been first used to calculate the low lying excitations spectrum of a harmonically trapped atomic bec in @xref . the result obtained from the variational method matches with stringari s result within the sum rule approach . in @xref , it is shown that the oscillation frequencies obtained from the exact ground state and a gaussian ansatz are in good agreement .
[95] one can use the time dependent variational approach to describe the vortex state .
[147] in these regimes , we have calculated the lower bound of the ground state energy , sound velocity , monopole and quadrupole mode frequencies .

GraphKvD[50]
39.73 34.40 3.69 192 [1] most of the properties of these dilute gas can be explained by considering only two body short range interaction which is characterized by the s wave scattering length .
[48] therefore , we expand around the time dependent variational parameters around the equilibrium widths in the following way , and @xmath . the time evolution of the widths around the equilibrium points are @xmath is the first order fluctuations around the equilibrium points of @xmath .
[95] one can use the time dependent variational approach to describe the vortex state .
[112] the vortex state play an important role in characterizing the superfluid properties of bose system .
[113] the critical angular frequency required to produce a vortex state is where is the energy of a vortex states with vortex quantum number and is the energy with no vortex . [147] in these regimes , we have calculated the lower bound of the ground state energy , sound velocity , monopole and quadrupole mode frequencies .
[68] the ground state energy per particle varies as @xmath .
[74] the monopole and quadrupole frequencies obtained from the variational approach are similar to the exact numerical values .
[82] the trap potential and -wave interaction can be neglected .
[84] the total ground state energy is @xmath .
[87] the ground state energy per particle varies as @xmath .
[95] one can use the time dependent variational approach to describe the vortex state .
[114] the critical angular frequency vs. the dimensionless scattering parameter is shown in fig.4 . [119] tf g regime : for large -wave scattering length , kinetic energy can be neglected .
[130] the monopole mode frequency for an ordinary atomic bec in the tf regime is independent of the vortex .
[142] the @xmath is also less than the monopole mode frequency in the vortex free condensate .
[150] in the tf regime of an ordinary atomic bec , the monopole and quadrupole mode frequencies are independent of the scattering length .