{'fraction_non_alphanumeric': 0.03587020264546817, 'fraction_numerical': 0.01348797810647032, 'mean_word_length': 4.91539795721719, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the Development of the RST Spanish Treebank

In this article we present the RST Spanish Treebank, the first corpus annotated with rhetorical relations for this language. We describe the characteristics of the corpus, the annotation criteria, the annotation procedure, the inter-annotator agreement, and other related aspects. Moreover, we show the interface that we have developed to carry out searches over the corpus' annotated texts.


Introduction
The Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) is a language independent theory based on the idea that a text can be segmented into Elementary Discourse Units (EDUs) linked by means of nucleus-satellite or multinuclear rhetorical relations. In the first case, the satellite gives additional information about the other one, the nucleus, on which it depends (ex. Result, Condition, Elaboration or Concession). In the second case, several elements, all nuclei, are connected at the same level, that is, there are no elements dependent on others and they all have the same importance with regard to the intentions of the author of the text (ex. Contrast, List, Joint or Sequence). The rhetorical analysis of a text by means of RST includes 3 phases: segmentation, detection of relations and building of hierarchical rhetorical trees. For more information about RST we recommend the original article of Mann and Thompson (1988), the web site of RST 1 and the RST review by Taboada and Mann (2006a).
RST has been used to develop several applications, like automatic summarization, information extraction (IE), text generation, question-answering, automatic translation, etc. (Taboada and Mann, 2006b). Nevertheless, most of these works have been developed for English, German or Portuguese. This is due to the fact that at present corpora annotated with RST relations are available only for these languages (for English: Carlson et al., 2002, Taboada andRenkema, 2008; for German: Stede, 2004; for Portuguese: Pardo et al., 2008) and there are automatic RST parsers for two of them (for English: Marcu, 2000; for Portuguese: Pardo et al., 2008) or automatic RST segmenters (for English: Tofiloski et al., 2009). Scientific community working on RST applied to Spanish is very small. For example, Bouayad-Agha et al. (2006) apply RST to text generation in several languages, Spanish among them. Da Cunha et al. (2007) develop a summarization system for medical texts in Spanish based on RST. Da Cunha and Iruskieta (2010) perform a contrastive analysis of Spanish and Basque texts. Romera (2004) analyzes coherence relations by means of RST in spoken Spanish. Taboada (2004) applies RST to analyze the resources used by speakers to elaborate conversations in English and Spanish.
We consider that it is necessary to build a Spanish corpus annotated by means of RST. This corpus should be useful for the development of a rhetorical parser for this language and several other applications related to computational linguistics, like those developed for other languages (automatic translation, automatic summarization, IE, etc.). And that is what we pretend to achieve with our work. We present the development of the RST Spanish Treebank, the first Spanish corpus annotated by means of RST.
In Section 2, we present the state of the art about RST annotated corpora. In Section 3, we explain the characteristics of the RST Spanish Treebank. In Section 4, we show the search interface we have developed. In Section 5, we establish some conclusions and future work.

State of the Art
The most known RST corpus is the RST Discourse Treebank, for English (Carlson et al., 2002a(Carlson et al., , 2002b. It includes 385 texts of the journalistic domain, extracted from the Penn Treebank (Marcus et al., 1993), such as cultural reviews, editorials, economy articles, etc. 347 texts are used as a learning corpus and 38 texts are used as a test corpus. It contains 176,389 words and 21,789 EDUs. 13.8% of the texts (that is, 53) were annotated by two people with a list of 78 relations. For annotation, the annotation tool RSTtool 2 (O'Donnell, 2000) was used, with some adaptations. The principal advantages of this corpus stand on the high number of annotated texts (for the moment it is the biggest RST corpus) and the clarity of the annotation method (specified in the annotation manual by . However, some drawbacks remain. The corpus is not free, it is not on-line and it only includes texts of one domain (journalistic).
For English there is also the Discourse Relations Reference Corpus (Taboada and Renkema, 2008). This corpus includes 65 texts (each one tagged by one annotator) of several types and from several sources: 21 articles from the Wall Street Journal extracted from the RST Discourse Treebank, 30 movies and books' reviews extracted from the epinions.com website, and 14 diverse texts, including letters, webs, magazine articles, newspaper editorials, etc. The tool used for annotation was also the RSTtool. The advantages of this corpus are that it is free and on-line, and it includes texts of several types and domains. The disadvantages are that the amount of texts is not very high, the annotation methodology is not 2 http://www.wagsoft.com/RSTTool/ specified and it does not include texts annotated by several people.
Another well-known corpus is the Potsdam Commentary Corpus, for German (Stede, 2004;Reitter and Stede, 2003). This corpus includes 173 texts on politics from the on-line newspaper Märkische Allgemeine Zeitung. It contains 32,962 words and 2,195 sentences. It is annotated with several data: morphology, syntax, rhetorical structure, connectors, correference and informative structure. Nevertheless, only a part of this corpus (10 texts), which the authors name "core corpus", is annotated with all this information. The texts were annotated with the RSTtool. This corpus has several advantages: it is annotated at different levels (the annotation of connectors is especially interesting); all the texts were annotated by two people (with a previous RST training phase); it is free for research purposes, and there is a tool for searching over the corpus (although it is not available on-line). The disadvantages are: the genre and domain of all the texts are the same, the methodology of annotation was quite intuitive (without a manual or specific criteria) and the inter-annotator agreement is not given.
For Portuguese, there are 2 corpora, built in order to develop a rhetorical parser (Pardo et al., 2008). The first one, the CorpusTCC (Pardo et al., 2008), was used as learning corpus for detection of linguistic patterns indicating rhetorical relations. It contains 100 introduction sections of computer science theses (53,000 words and 1,350 sentences). To annotate the corpus a list of 32 rhetorical relations was used. The annotation manual by  was adapted to Portuguese. The annotation tool was the ISI RST Annotation Tool 3 , an extension of the RSTtool. The advantages of this corpus are: it is free, it contains an acceptable number of texts and words and it follows a specific annotation methodology. The disadvantage is: it only includes texts of one genre and domain, only annotated by one person.
The second one, Rhetalho (Pardo and Seno, 2005), was used as reference corpus for the parser evaluation. It contains 50 texts: 20 introduction sections and 10 conclusion sections from computer science scientific articles, and 20 texts from the online newspaper Folha de São Paulo (7 from the Daily section, 7 from the World section and 6 from the Science section). It includes approximately 5,000 words. The relations and the annotation tool are the same as those used in the CorpusTCC. The advantages of this corpus are that it is free, it was annotated by 2 people (they both were RST experts and followed an annotation manual) and it contains texts of several genres and domains. The main disadvantage is the scarce amount of texts.
The Penn Discourse Treebank (Rashmi et al., 2008)f for English includes texts annotated with information related to discourse structure and semantics (without a specific theoretical approach). Its advantages are: its big size (it contains 40,600 annotated discourse relations) allows to apply machine learning, and the discourse annotations are aligned with the syntactic constituency annotations of the Penn Treebank. Its limitations are: dependencies across relations are not marked, it only includes texts of the journalistic domain, and it is not free. Although there are several corpora annotated with discourse relations, there is not a corpus of this type for Spanish.

The RST Spanish Treebank
As Sierra (2008) states, a corpus consists of a compilation of a set of written and/or spoken texts sharing some characteristics, created for certain investigation purposes. According to Hovy (2010), we use 7 core questions in corpus design, detailed in the next subsections.

Selecting a Corpus
For the RST Spanish Treebank, we wanted to include short texts (finally, the average is 197 words by text; the longest containing 1,051 words and the shortest, 25) in order to get a best on-line visualization of the RST trees. Moreover, in the first stage of the project, we preferred to select specialized texts of very different areas, although in the future we plan to include also nonspecialized texts (ex. blogs, news, websites) in order to guarantee the representativity of the corpus. We did not find a pre-existing Spanish corpus with these characteristics, so we decided to build our own corpus. Following Cabré (1999), we consider that a text is specialized if it is written by a professional in a given domain. According to this work, specialized texts can be divided in three levels: high (both the author and the potential reader of the text are specialists), average (the author of the text is a specialist, and the potential reader of that text is a student or someone interested in or possessing some prior knowledge about the subject) and low (the author of the text is a specialist, and the potential reader is the general public). The RST Spanish Treebank includes specialized texts of the three mentioned levels: high (scientific articles, conference proceedings, doctoral theses, etc.), average (textbooks) and low (articles and reports from popular magazines, associations' websites, etc.). The texts have been divided in 9 domains (some of them including subdivisions): Astrophysics, Earthquake Engineering, Economy, Law, Linguistics (Applied Linguistics, Language Acquisition, PLN, Terminology), Mathematics (Primary Education, Secondary Education, Scientific Articles), Medicine (Administration of Health Services, Oncology, Orthopedy), Psychology and Sexuality (Clinical Perspective, Psychological Perspective).
The size of a corpus is also a polemic question. If the corpus is developed for machine learning, its size will be enough when the application we want to develop obtains acceptable percentages of precision and recall (in the context of that application). Nevertheless, if the corpus is built with descriptive purposes, it is difficult to determine the corpus size. In the case of a corpus annotated with rhetorical relations, it is even more difficult, because there are various factors involved: EDUs, SPANs (that is, a group of related EDUs), nuclearity and relations. In addition, relations are multiple (we use 28). As Hovy (2010: 13) mentions, one of the most difficult phenomena to annotate is the discourse structure. Our corpus contains 52,746 words and 267 texts. Table 1 includes RST Spanish Treebank statistics in terms of texts, words, sentences and EDUs.  To increase the linear performance of a statistical method, it is necessary that the training corpus size grows exponentially (Zhao et al., 2010). However, the RST Spanish Treebank is not designed only to use statistical methods; we think it will be useful to employ symbolic or hybrid algorithms (combining symbolic and statistical methods). Moreover, this corpus will be dynamic, so we expect to have a bigger corpus in the future, useful to apply machine learning methods.
If we measure the corpus size in terms of words or texts, we can take as a reference the other RST corpora. Nevertheless, as Sierra states (2008), it is "absurd" to try to build an exhaustive corpus covering all the aspects of a language. On the contrary, the linguist looks for the representativeness of the texts, that is, tries to create a sample of the studied language, selecting examples which represent the linguistic reality, in order to analyze them in a pertinent way. In this sense and in the frame of this work, we consider that the size will be adequate if the rhetorical trees of the corpus include a representative number of examples of rhetorical relations, at least 20 examples of each one (taking into account that the corpus contains 3115 relations, we consider that this quantity is acceptable; however, we expect to have even more examples when the corpus grows). Table 2 shows the number of examples of each relation currently included into the RST Spanish Treebank (N-S: nucleus-satellite relation; N-N: multinuclear relation). As it can be observed, it contains more than 20 examples of most of the relations. The exceptions are the nucleus-satellite relations of Enablement, Evaluation, Summary, Otherwise and Unless, and the multinuclear relations of Conjunction and Disjunction, because it is not so usual to find these rhetorical relations in the language, in comparison with others. Hovy (2010: 128) states that, given the lack of examples in the corpus, there are 2 possible strategies: a) to leave the corpus as it is, with few or no examples of some cases (but the problem will be the lack of training examples for machine learning systems), or b) to add low-frequency examples artificially to "enrich" the corpus (but the problem will be the distortion of the native frequency distribution and perhaps the confusion of machine learning systems). In the current state of our project, we have chosen the first option. We think that, including specialized texts in a second stage, we will get more examples of these less common relations. If we carry out a more granulated segmentation maybe we could obtain more examples; however, we wanted to employ the segmentation criteria used to develop the Spanish RST discourse segmenter (da Cunha et al., 2011).

Instantiating the Theory
Our segmentation and annotation criteria are very similar to the original ones used by Mann and Thompson (1988) for English, and by da Cunha and Iruskieta (2010) for Spanish. We also explore the annotation manual for English by Carlon and . Though we use some of their postulates, we think that their analysis is too meticulous in some aspects. Because of this, we consider that it is not adjusted to our interest, which is the finding of the simplest and most objective annotation method, orientated to the future development of a rhetorical parser for Spanish. To sum up, our segmentation criteria are: a) All the sentences of the text are segmented as EDUs (we consider that a sentence is a textual passage between a period and another period, a semicolon, a question mark or an exclamation point; texts' titles are also segmented [Se muestra que el modelo discreto en diferencias finitas es convergente y que su realización se reduce a resolver una sucesión de sistemas lineales tridiagonales.] [It appears that the discreet model in finite differences is convergent and that its accomplishment is to solve a succession of tridiagonal linear systems.] b3) Subordinate relative clauses are not segmented. Ex.
[Durante el proceso, que utiliza solo aritmética entera, se obtiene el determinante de la matriz de coeficientes del sistema, sin necesidad de cálculos adicionales.] [During the process, which only uses entire arithmetic, the determinant of the system coefficient matrix is obtained, without additional calculations.] b4) Elements in parentheses are only segmented if they follow the criterion b1. Ex.
[Este año se cumple el bicentenario del nacimiento de Niels (Nicolás, en nuestro idioma) Henrik Abel.] [This year is the bicentenary of Niels's birth (Nicolás, in our language) Henrik Abel.] b5) Embedded units are segmented by means of the non-relation Same-Unit proposed by Carlon and . Figure 1 shows Figure 1: Example of the non-relation Same-Unit

Designing the Interface
The annotation tool used in this work is the RSTtool, since it is free and easy to use. Therefore, we preferred to use it instead of designing a new one. Nevertheless, we have designed an on-line interface to include the corpus and to carry out searches over it (see Section 4).

Selecting and Training the Annotators
With regard to the corpus annotators, we have a team of 10 people (last year Bachelor's degree students, Master's degree students and PhDs) 5 . Before the annotation, they took a RST course of 6 months (100 hours), where the segmentation and annotation methodology used for the development of the RST Spanish Treebank was explained. 6 We called this period "training phase". The course had a theoretical and a practical part. In the theoretical part, some criteria with regard to the 3 phases of rhetorical analysis (segmentation, detection of relations, and rhetorical trees building) were given to annotators. In the practical part, firstly, it was explained how to use the RSTtool. Secondly, annotators extracted several texts from the web, following their personal interests, as for example, music, video games, cookery or art webs. They segmented those texts, using the established segmentation criteria. Once segmented, all the doubts and problematic examples were discussed, and they tried to get an agreement on the most complicated cases. Thirdly, the relations were analyzed (using a given relations list) and, once again, annotators discussed the difficult cases. After the discussion, texts were re-annotated to verify if the difficulties were solved. This process was doubly interesting, since it helped to create common criteria for the annotation of the final corpus and to define the annotation criteria more clearly and consensually, in order to include them in the RST Spanish Treebank annotation manual. Once annotators agreed on the most difficult cases, we consider that the training phase finished.

Procedure
We start from the following annotation definition: Annotation ('tagging') is the process of adding new information into source material by humans (annotators) or suitably trained machines. [...]. The addition process usually requires some sort of mental decision that depends both on the source material and on some theory or knowledge that the annotator has internalized earlier. (Hovy, 2010: 6) Exactly, after our annotators internalized the theory and annotation criteria during the training phase, the "annotation phase" of the final texts included in the RST Spanish Treebank started. In this phase, the annotation tasks were assigned to annotators (the number of texts assigned to each annotator was different, depending on their availability). They were asked to carry out the annotation individually and without questions among them. We calculated that the average time to carry out the annotation of one text was between 15 minutes and 1 hour. This time difference is due to the fact that the corpus includes both short and long texts. The annotation process is the following: once a text is segmented, rhetorical relations between EDUs are annotated. First, EDUs inside the same sentence are annotated in a binary way. Second, sentences inside the same paragraph are linked. Finally, paragraphs are linked. Hovy (2010) states that it is difficult to determine if, for the same money (we add "for the same time"), it is better to double-annotate less, or to single-annotate more. As he explains, Dligach et al. (2010) made an experiment with OntoNotes (Pradhan et al., 2007) verb sense annotation. The result was that, assuming the annotation is stable (that is, inter-annotator agreement is high), it is better to annotate more, even with only one annotator. The problem with RST annotation is that there are so many categories to annotate, that is very difficult to obtain a stable annotation. Therefore, we consider it is necessary to have at least some texts double-annotated (or even tripleannotated), in order to have an adequate discourse corpus. This is the reason why, following the RST Discourse Treebank methodology, we use some texts as learning corpus and some others (from the Mathematics, Psychology and Sexuality domains) as test corpus: 69% (183 texts) and 31% (84 texts), respectively. The texts of the learning corpus were annotated by 1 person, whereas the texts of the test corpus were annotated by 2 people.

Validating Results
Da Cunha and Iruskieta (2010) measure interannotator agreement by using the RST trees comparison methodology by Marcu (2000). This methodology evaluates the agreement on 4 elements (EDUs, SPANs, Nuclearity and Relations), by means of precision and recall measures (an annotation with regard to the other one). Following this methodology, we have measured inter-annotator agreement over the test corpus. We employ an on-line automatic tool for RST trees comparison, RSTeval (Mazeiro and Pardo, 2009), where Marcu's methodology has been implemented (for 4 languages: English, Portuguese, Spanish and Basque). We know that there are some other ways to measure agreement, such as Cohen's kappa (Cohen, 1960) or Fleiss's kappa (Fleiss, 1971), for example. Nevertheless, we consider that Marcu's methodology (2000) is suitable to compare adequately 2 annotations of the same original text, because it has been designed specifically for this task.
For each trees pair from the test corpus, precision and recall were measured separately. Afterwards, all those individual results were put together to obtain general results. Table 3 shows global results for the 4 categories. The category with more agreement was EDUs (recall: 91.04% / precision: 87.20%), that is, segmentation. This result was expected, since the segmentation criteria given to the annotators were quite precise and the possibility of mistake was low. The lowest agreement was obtained for the category Relations (recall: 78.48% / precision: 76.81%). This result is lower than the other, but we think it is acceptable. In the RST Discourse Treebank the trend was similar to the one detected in our corpus: the highest agreement is obtained at the segmentation level and the lowest at the relations level.  Table 3: Inter-annotator agreement Precision and recall have not been calculated with respect to a gold standard because it does not exist for Spanish. Our future aim is to reach a consensus on the annotation of the test corpus (using an external "judge"), in order to establish a set of texts considered as a preliminary gold standard for this language. We consider that the annotations have quality at present, because interannotator agreement is quite high; however, this consensus could solve the typical annotation mistakes we have detected or some ambiguities.
We have analyzed the main discrepancy reasons between annotators. With regard to the segmentation, the main one was human mistake; ex. segmenting EDUs without a verb (one annotator segmented the following passage into 2 EDUs because she detected a Means relation, but the second EDU does not include any verb): [Además estudiamos el desarrollo de criterios para determinar si un semigrupo dado tiene dicha propiedad ] [mediante el estudio de desigualdades de curvaturadimensión. ] [We also study the development of tests in order to determine if a given semi group has this property] [by means of curvature-dimension inequalities.] The second reason was that in the manual some aspects were not explained in detail. For example, if a substantive sentence or a direct/object clause (which must not be segmented, according to the point b2) includes two coordinated clauses, these must not be segmented either. Thus, we found some erroneous segmentations. For example: [Los hombres adultos tienen miedo de fracasar] [y no cumplir con el rol masculino de ser proveedores del hogar y de proteger a su familia.] This kind of mistakes allowed us to refine our segmentation manual a posteriori. In the future, we will ask the test corpus annotators to make a new annotation of the texts, using the refined manual, in order to check if the agreement increases, in the same way as the RST Discourse Treebank.
With regard to rhetorical annotations, we detected 2 main reasons of inter-annotator disagreement. The first one was the ambiguity of some relations and their corresponding connectors; for example, Justification-Reason, Antithesis-Concession or Circumstance-Means relations, like in the following passage (in Spanish, "al" may indicate time or manner): The second one is due to differences between annotators when determining nuclearity. For example, in the following passage, one annotator marked Background and the other one Elaboration: It is easier to solve segmentation disagreement than relations disagreement, since in this case annotator subjectivity is more evident; we must consider how to refine our manual in this sense.

Delivering and Maintaining the Product
Hovy (2010) mentions some technical issues regarding these points: licensing, distribution, maintenance and updates. With regard to licensing and distribution, the RST Spanish Treebank will be free for research purposes. We have a data manager responsible for maintenance and updates.
The description of the annotated corpus is also a very important issue (Ide and Pustejovsky, 2010). It is important to provide a high level description of the corpus, including the theoretical framework, the methodology (annotators, annotation manual and tool, agreement, etc.), the means for resource maintenance, the technical aspects, the project leader, the contact, the team, etc. The RST Spanish Treebank includes all this detailed information.
XML (with a DTD) has been used, in order the corpus can be reused for several aplications. In the future, we plan to use the standard XCES.
To know more about resources development, linguistic annotation or inter-annotator agreement, we recommend: Palmer et al. (on-line), Palmer and Xue (2010), and Artstein and Poesio (2008).

The Search Interface of the RST Spanish Treebank
The RST Spanish Treebank interface is freely available on-line 7 . It allows the visualization and downloading of all the texts in txt format, with their corresponding annotated trees in RSTtool format (rs3), as well as in image format (jpg). Each text includes its title, its reference, its web link (if it is an on-line text) and its number of words. The interface shows texts by areas and allows the user to select a subcorpus (including individual files or folders containing several files). The selected subcorpus can be saved on local disk (generating a xml file) for future analyses. The interface includes a statistical tool which allows obtaining statistics of rhetorical relations in a subcorpus selected by the user. The RSTtool also offers this option but it can be only used for one text. We consider that it is more useful for the user to obtain statistics from various texts, in order to get significant statistical results. As the RSTtool, our tool allows to count the multinuclear relations in two ways: a) one unit for each detected multinuclear relation, and b) one unit for each detected nucleus. If we use b), the statistics of the multinuclear relations of Table 2 are higher: List (864), Joint (537), Sequence (289), Contrast (153), Conjunction (28) and Disjunction (24).
We are developing another tool, aimed to extract information from the annotated texts, which we will soon include into the interface. This tool will allow to the user to select a subcorpus and to extract from it the EDUs corresponding to the rhetorical relations selected, like a multidocument specialized summarizer guided by user's interests.
The RST Spanish Treebank interface also includes a screen which permits the users to send their own annotated texts. Our aim is for the RST Spanish Treebank to become a dynamic corpus, in constant evolution, being increased with texts annotated by users. This has a double advantage since, on the one hand, the corpus will grow and, on the other hand, users will profit from the 7 http://www.corpus.unam.mx/rst/ interface's applications, using their own subcorpora. The only requirement is to use the relations and the segmentation and annotation criteria of our project. Once the texts are sent, the RST Spanish Treebank data manager will verify if the annotation corresponds to these criteria.

Conclusions and Future Work
We think that this work means an important step for the RST research in Spanish, and that the RST Spanish Treebank will be useful to carry out diverse researches about RST in this language, from a descriptive point of view (ex. analysis of texts from different domains or genres) and an applied point of view (development of discourse parsers and NLP applications, like automatic summarization, automatic translation, IE, etc.).
For the moment the corpus' size is acceptable and, though the percentage of double-annotated texts is not very high, we think that having 10 annotators (using the same annotation manual) avoids the bias of only one annotator. In addition, the corpus includes texts of diverse domains and genres, which provides us with a heterogeneous Spanish corpus. Moreover, the corpus interface that we have designed allows the user to select a subcorpus and to analyze it statistically. In addition, we think that it is essential to release a free corpus, on-line and dynamic, that is, in continuous growth. Nevertheless, we are conscious that our work still has certain limitations, which we will try to solve in the future. In the short term, we have 5 aims: a) To add one more annotator for the test corpus and to measure inter-annotator agreement. b) To use more agreement measures, like kappa. c) To reach a consensus on the annotation of the test corpus, in order to establish a set of texts considered as a preliminary gold standard. d) To finish and to evaluate the IE tool. e) To analyze the corpus to extract linguistic patterns for the automatic relations detection.
In the long term, we consider other aims: f) To increase the corpus, by adding nonspecialized texts, and new domains and genres. g) To annotate all the texts by 3 people, to get a representative gold-standard for Spanish (this aim will depend on the funding of the project).