{'fraction_non_alphanumeric': 0.025744780670889044, 'fraction_numerical': 0.009500351864883884, 'mean_word_length': 4.315773067331671, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
RALI System Description for CL-SciSumm 2016 Shared Task

We present our approach to the CL-SciSumm 2016 shared task. We propose a technique to determine the discourse role of a sentence. We differentiate between words linked to the topic of the paper and the ones that link to the facet of the scientific discourse. Using that information, histograms are built over the training data to infer a facet for each sentence of the paper (result, method, aim, implication and hypothesis). This helps us identify the sentences best representing a citation of the same facet. We use this information to build a structured summary of the paper as an HTML page.


Introduction
One's task in research is to read scientific papers to be able to compare them, to identify new problems, to position a work within the current literature and to elaborate new research propositions [8].
This implies reading many papers before finding the ones we are looking for. With the growing amount of publications, this task is getting harder. It is becoming important to have a fast way of determining the utility of a paper for our needs. A first solution is to use web sites such as CiteSeer, arXiv, Google Scholar and Microsoft Academic Search that provide cross reference citations to papers. Another approach is automatic summarization of a group of scientific papers dealing with the subject.
This year's CL-SciSumm competition for summarization of computational linguistics papers proposes a community approach to summarization; it is based on the assumption that citances, the set of citation sentences to a reference paper, can be used as a measure of its impact. This task implies identifying the text a citance refers to in the reference paper and a facet (aim, result, method, implication and hypothesis) for the referred text.
We are building a system that given a topic, generates a survey of the topic from a set of papers. That system uses citations as the primary source of information for building an annotated summary. Our system must be able to identify the purpose/polarity/facet of a citation. to direct the reader towards the more relevant information. The summary is built by selecting sentences from the cited paper and the citations. This process uses a similarity function between sentences. The resulting summaries are presented in HTML format with their annotations and links to the original paper. The only task that is not performed by our system is finding the text referred to by the citation. We intend to use the information already found by our system (facet of citations and sentences) to complete that task.
We had already some experience in dealing with scientific papers and their references, having participated to Task2 of the Semantic Publishing Challenge of ESWC-2014 (Extended Semantic Web Conference) on the extraction and characterization of citations. A short review of previous work follows in Sect. 2. We will summarize the task in Sect. 3 and the techniques for extracting information in Sect. 4. Finally, Sect. 5 will show our results.

Previous Work
There has been a growing attention towards the information carried by citations and their surrounding sentences (citances). These contain information useful for rhetorical classification [18], technical surveys [14] and emphasize the impact of papers [12]. Qazvinian [16] and Elkiss [6] showed that citations provide information not present in the abstract.
Since the first works of Luhn [11] and Edmundson [5] many researchers have developed methods for finding the most relevant sentences of papers to produce abstracts and summaries. Many metrics have been introduced to measure the relevance of parts of text, either using special purpose formulas [21] or using learned weights [10]. The hypothesis for CL-SciSumm task is that important sentences can be pointed out by other papers : a citation indicates a paper considered important by the author of the citing paper.
Another domain for study over scientific papers is the classification of their sentences. Teufel [19] identified the rhetorical status of sentences using Bayes classifier.
To find citations inside a paper, we need to analyse the references section. Dominique Besagni et al. [1] developed a method using pattern recognition to extract fields from the references while Brett Powley and Robert Dale [15] looked citations and references simultaneously using informations from one task to help complete the second task.

Task Description
For this year competition we were given 30 topics, 10 for training, 10 for tuning and 10 for testing [9]. Each topic is composed of a Reference Paper (RP) and some Citing Papers (CPs). The citing papers contain citations pointing to the RP. An annotation file is given for each topic. That file contains information about each citation, the citation marker and the citance.
There are two mandatory tasks (Task 1A and Task 1B) and an optional task (Task 2) 3 .
Task 1A : Find the part of the RP that is indicated with each citance. This will be called the referenced text.
Task 1B : Once the referenced text is identified, we need to attribute a facet to it. A facet is one of these : result, method, aim, implication and hypothesis. Task 2 : Building a summary for the RP using the referenced text identified in Task 1A.
Both the training and the developing set of topics contain expected results for these tasks. The next section will describe how our system performs on the test set.

Our Approach
For the first task, we have to find the referenced text and its facet. We hypothesized that the referenced text should be sentences sharing the same facet as the citance. We use that fact to reduce the set of sentences to choose from for the reference. This is why we execute Task 1B on all the sentences of the RP and all the citances prior to Task 1A. We now present how we determine the facet of a citance, then the facet for sentences in the RP and finally the referenced text.

Task 1B : Facet Identification
Our goal is to be able to use our system for papers from different domains, without having to train them again. Toward that objective, our system only uses words that are not domain specific. Patrick Drouin [3,4] compiled such a list of words in his Transdisciplinary scientific lexicon (TSL). This lexicon comprises 1627 words such as acceptance, gather, newly, severe... We will denote the set of words from the lexicon using w ∈ L.
We trained two systems, one to attribute a facet to sentences in the RP and one to attribute a facet to citances.
We determine the word distribution for each facet using an histogram. We only use words appearing in the TSL. This computation yielded a sum of each words present in all referenced text for each facet. The facet with the highest score is chosen for that sentence.
For training our system, we extract the reference sentences from each annotation with their assigned facet. Each sentence is tokenized using the NLTK library in Python. Only words from the TSL are kept. Our dataset consists of pairs of list of words with a facet : We build a profile (h f ) for each facet using a histogram. For each word in the lexicon, we compute the number of times it appears in sentence paired with the a specific facet.
When a word appears more then once in a sentence, all its occurrences are counted. Once the histogram is built, we use it to find the facet of new sentence. First, we extract the words that are part of the lexicon from the sentence, yielding the list of words p. Then a score s f for each facet is computed by adding the profile of each word for that facet. The facet that scored the highest value is assigned to the new sentence.
Looking closely at the results for the profile, we saw that some words have a negative effect on finding the facet. To find a better sublist of words to use within the TSL, we used a genetic algorithm that uses a population of lists of words.
A genetic algorithm starts with an initial population (set of possible solutions) and tries to find better solutions by applying small changes to existing solutions. In our case, a solution is a subset of words L i of L. The initial population is built using random subsets.
To build the next generation, we use three different techniques : 1. Adding a random word to an existing solution : Removing a random word from an existing solution : Combining two subsets of existing solutions : Once enough solutions are built for the new population, each solution is tested using cross-validation with the histogram. The list that performed best in the task is kept for the next generation. We use the same technique over the dataset consisting of the citance texts and their facets.

Task 1A : Finding the Sentences Referred to by Citances
Having determined the facet of sentences in both the RP and citances, we are now ready to assign referenced text to citances from the CPs. Our hypothesis is that a citance should have the same facet as the text it refers to. We extract Q f the subset of sentences from the RP that have the same facet f as a citance c i . To choose the sentence of RP referred to by the citance, we look for the sentence from Q f that is the most similar with the citance c i . sim mcs (P 1 , P 2 )= 1 2 (hs(P 1 , P 2 ) + hs(P 2 , P 1 )) (1) We use the similarity function sim mcs defined by Mihalcea, Corley and Strapparava [13]. This similarity function between sentences P 1 and P 2 (Equation 1) averages two values, the similarity from P 1 to P 2 , and the similarity from P 2 to P 1 . The similarity from one sentence P i to the other P j is computed by first pairing each word from the first sentence w ∈ P i with a word in the second one v ∈ P j . A word is paired with the one that is the most similar to it (Equation 3). For each pair (w, v), the value of the similarity is weighted by the Inverse Document Frequency of the first word idf w (Equation 2). The average of these weighted similarity values is computed to yield the similarity between P i and P j . We use only words that are Noun, Verb, Adjective and Adverb for this comparison. The POS_tagger of NLTK was used to assert the tag of each word. Since we believe that the domain of the paper is important to compute that similarity, we use all words, not only the ones that are part of the TSL.
Mihalcea et al. [13] reported that within the set of possible metrics to compare words, the one proposed by Zhibiao Wu et Martha Palmer [20] yielded good result (denoted sim wup ). This metric is also available with the NLTK package. To use that metric, we transform each word into their synonym group synset using WordNet. The IDF was computed for each synset. The computation was done over the set of all the documents contained in the ACL Anthology Network 4 .

Task 2 : Summarization
Multiple source summarization adds three problems [17] : 1. Redundancy : a paper will often be cited for the same reason over and over, resulting in many citances having the same subject. 2. Identifying important differences between sources : our goal will be to find those citances/references that bring new information and important information to the summary. 3. Coherence : since sentences come from many sources, we want to ensure that the summary forms an unified whole.
For Task 2, we choose to use the Maximal Marginal Relevance (MMR) proposed by Jaime G. Carbonell et Jade Goldstein [2]. Their technique is presented in Equation 4, in which R is the list of possible sentences and V is the summary. They propose to use the title of the research paper as the starting query Q.
At each iteration, their algorithm adds a sentence s i to V . Sentences are choosen so that they bring new information to the summary (Points 1 and 2) and it must have a certain amount of similarity with the query (Point 2). λ must be adjusted to balance between adding a sentence very similar to the query and a sentence very different from the ones already in the summary V . We use the same metric (sim mcs ) as for task 1A to compare sentences.
We divided the summarization process in two steps : adding sentences from the citance (R = CT) and adding sentences from the paper (R = RP). In the first step, the algorithm chooses sentences from the set of citances until it reaches 150 words. For that part, we use λ = 0.3 to give priority to similarity with the query, trying to remove meaningless citances. Since citances have been identified as bringing new information not present in the original paper, we believe it is important to keep them in the summary. Then, the summary is completed (to 250 words) using sentences choosen from the RP. Here, we use λ = 0.7. Since sentences are choosen in the RP, most of them are about the same subject, we want to give priority to sentences that are more different.
The summary is built in an XML format. Each sentence is identified with its position (the id of the paper it was extracted from, the sid and ssid attributes inside the XML source files). The citances contain the id of the referred paper. This information will enable to point a reader towards the corresponding paper.
To help analyse the summaries, our software builds an HTML page containing the extracted information (see Fig. 1).

Task 1
We present our results for facet attribution to citance and reference text. The set of data we receive is divided in two : the training set contains 197 sentences distributed over the citances and 247 sentences over the reference text; the development set contains 273 sentences distributed over the citances and 330 sentences over the reference text. We first train our system using the training data (T) and then we retrained it using both set training and developing set together (TD). In each case, we test the result over both sets. We show the result for simple training of the histogram and for the training using the genetic algorithm (gen_T) to select the list of words to consider. We also trained our histogram without limiting to the words in the TSL for comparison purpose.
For the genetic algorithm, we let it run over 25 generations. Each generation started with 1 000 lists of words. 9 000 lists are added using the proposed mutations, bringing the number of lists to 10 000. The result of these experiments are presented in Table 1 and Table 2. We see that, using the training set T gives good result on itself but lower result when we apply it on the development set. After training with both set TD (Test + Development), the result over the development set raises at the expense of the result for the training set. For citance, the genetic algorithm yields better result over the training set only. It does not help to get better histograms. Considering that fact, we ask ourselves if it is possible to Once we had identified the facets, we ran our script for finding the reference text. It was able to reach an F1 score of 0.095 over the training set and 0.052 over the development set (table 3). We reduced the search space for the referred text using the facet of the citance. Since the identification of the facet is not perfect, this reduction might remove a sentence we are looking for. In the future, we have to test our approach with all sentences, instead of the reduced set, to see if this reduction of space causes a problem more than helps the solution.  Figure 1 shows the HTML interface we have generated for showing the result of our system. It allows for selecting different topics. The top of the page lets us choose between the different topics that were summarised. Each topic will present, on the left side, the text of each CPs and RP. The sentences have been divided and citance identified. The right side contains the different summaries that our software builds (using different values of λ) and the gold standard summary. Each paper links to its pdf version on the ACL Anthology 5 .

Task 2
On the left side of the top part of the figure we see the RP divided in sentences. On the right side, there is a summary built by choosing five sentences from the set of citances using a λ of 0.3. These sentences where selected to be as different as possible by the MMR algorithm. The bottom screen shoot (Fig. 1) presents one of the CP on the left. The citance and citation are colored to be easy to identify. The third sentence from the top was selected by the algorithm for the summaries.

Conclusion
We presented the use of distinguishing between topic and non-topic (TSL) words for determining the facet of sentences in a paper. This technique is useful because it lets our system work on paper in a domain independent way. We obtained good results with a simple histogram. We still have to test our histogram over other domains, to see if they also yield good results. Our experiments with a genetic algorithm to refine the list of used words did not show any improvement.
We presented our interface for browsing the results of our system. That interface presents RP, CPs and summaries with links to the original paper. This interface helps the reader browse through a topic.