{'fraction_non_alphanumeric': 0.02978790589968949, 'fraction_numerical': 0.011209936319141097, 'mean_word_length': 5.177503250975293, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A contrastive review of paraphrase acquisition techniques

This paper addresses the issue of what approach should be used for building a corpus of sententential paraphrases depending on one's requirements. Six strategies are studied: (1) multiple translations into a single language from another language; (2) multiple translations into a single language from different other languages; (3) multiple descriptions of short videos; (4) multiple subtitles for the same language; (5) headlines for similar news articles; and (6) sub-sentential paraphrasing in the context of a Web-based game. We report results on French for 50 paraphrase pairs collected for all these strategies, where corpora were manually aligned at the finest possible level to define oracle performance in terms of accessible sub-sentential paraphrases. The differences observed will be used as criteria for motivating the choice of a given approach before attempting to build a new paraphrase corpus.


Introduction
Paraphrases constitute admittedly one of the basic types of knowledge required by Natural Language Processing systems, and automatic acquisition has attracted a lot of work (Madnani and Dorr, 2010). One of the most common approaches for acquiring sub-sentential paraphrases is to first collect pairs of sentences. If these pairs are paraphrases, then it is expected that they contain many precise sub-sentential paraphrases. Otherwise, the nature of the sentence pairs will have an impact on the numbers and types of sub-sentential paraphrases that they contain. In this article, we address this issue on French by considering a large range of possible strategies for collecting sentence pairs, and performing an annotation study to describe the sub-sentential paraphrases that are found in each corpus type. The objective of this paper can be paraphrased as that of providing information for selecting a type of corpus depending on specific requirements for paraphrase acquisition.
We will first briefly review the domain of paraphrase corpus construction (section 2.). We will then describe the 6 strategies that we have studied for this work (section 3.): (1) multiple translations into a single language from another language; (2) multiple translations into a single language from different other languages; (3) multiple descriptions of short videos; (4) multiple subtitles for the same language; (5) headlines for similar news articles; and (6) sub-sentential paraphrasing in the context of a Web-based game. An experiment were 50 sentence pairs for each corpus types were manually annotated will then be reported (section 4.1.). Annotation was done at the finest possible level to define oracle performance in terms of accessible sub-sentential paraphrases. We will describe a typology of sub-sentential paraphrases per corpus type (section 4.2.) and conclude (section 5.). The differences observed may be used in future research as criteria for motivating the choice of a given approach before attempting to build a new paraphrase corpus.

Building paraphrase corpora
Finding equivalent text units at different levels (lexical, phrasal or sentential) can be beneficial to a number of NLP tasks, such as Information Retrieval (Pasça, 2005), Machine Translation (Schroeder et al., 2009), and Summarization (Hirao et al., 2004). We briefly review here the main approaches for building paraphrase corpora. Several works have used monolingual parallel corpora obtained by multiple human translations of the same foreign source text. For instance, Barzilay and McKeown (2001) exploited monolingual parallel corpus of multiplytranslated novels and used contextual information based on lexical similarity to extract paraphrases. The corpora described by Cohn et al. (2008) include sentential paraphrases acquired from multiply-translated Chinese news sentences into English. Because this type of resource is extremely scarce, techniques based on monolingual comparable corpora have been proposed, as this type of resource is much more readily available. Such corpora often correspond to texts grouped with respect to common subject and time frame. Barzilay and Elhadad (2003) learned rules that exploit topic structure and local alignment to extract sentence pairs from news stories about the same events from different press agencies. Dolan and Brockett (2005) built the Microsoft Research Paraphrase Corpus, which contains sentence pairs collected using heuristic extraction techniques and a SVM classifier to select likely sentential paraphrases from a large corpus of thematically clustered news texts. An in-depth review of these methods is given in (Madnani and Dorr, 2010). Provided corpora are large enough, paraphrase acquisition techniques can potentially extract a large number of subsentential paraphrases. However, corpora may have genres and/or domains that are not adapted to specific requirements. Furthermore, it may be the case that some useful textual units are never observed in such corpora. Collaborative resource creation has therefore been used to overcome in part those difficulties. Paraphrase acquisition has for instance been formulated in the context of a Web-based game (Chklovski, 2005), or as independent descriptions of short videos through crowdsourcing (Chen and Dolan, 2011). Other works have also considered looking at histories of collaboratively authored text such as the Wikipedia encyclopedia to identify rewritings corresponding to paraphrases (Max and Wisniewski, 2010). Lastly, the field of automatic paraphrase generation (Madnani and Dorr, 2010) has produced numerous techniques that have been used for generating sentential paraphrases used for learning Statistical Machine Translation systems (Nakov, 2008) or optimizing their parameters (Madnani et al., 2008), and various potential uses including uses by humans (Quirk et al., 2004;Zhao et al., 2010). In addition to the fact that the current performance of these techniques is limited, it should be noted that an issue of circular dependency arises, as these techniques rely on the availability of paraphrase resources; most of these techniques are, however, able to produce paraphrases that did not belong to their original resources.

Corpus types
We now describe the various corpus types that we have considered in our study. We have selected 6 strategies for collecting sentential paraphrase pairs, which, as our analyses will show, correspond to various degrees of sentence parallelism, and therefore allow to extract different numbers of sub-sentential paraphrases. Examples for each collected corpus types appear in Table 1. 1 3.1. Multiple translations from a single language or from different languages Multiple independent human translations of the same texts produce sentential paraphrases (Mitsuo Shimohata and Matsumoto, 2004), which can subsequently be used for sub-sentential paraphrase acquisition (Barzilay and McKeown, 2001). We built two sub-corpora from the MUL-TITRAD corpus (Bouamor, 2010), a paraphrase corpus for French built by collecting multiple translations from several languages proposed by volunteers. We selected a set of 50 sentences on the basis that 4 independent valid translations from English and one valid translation from German, Spanish, Italian and Portuguese were available. In each group, one paraphrase was randomly selected as a "reference paraphrase" which was paired to the three others. Two sub-corpora were built, one from a single language (English), and another one from different languages (German, Spanish, Italian and Portuguese). The first two rows of Table 1 illustrate examples from those two corpora. The last line in italics is the reference sentence in English, which has been translated twice into French. This same sentence, which was also available in German, Spanish, Italian and Portuguese, was also translated into French as shown in the examples of the second row of Table 1.

Multiple translations of subtitles
Many subtitles for movies and TV series are contributed by Web users in many languages, and several users sometimes independently contribute subtitles for the same video in the same language. We used the sentence alignment procedure that exploits time frames described in (Tiedemann, 2007), and collected a monolingual corpus for 50 sentence pairs of two subtitles of the TV series Desperate housewifes. The third row of Table 1 illustrates a sentence uttered in English (in italics) and its two proposed subtitles in French.

Multiple descriptions of Videos
Another type of semantic content for acquiring paraphrases is the scene recorded on a video. Chen and Dolan (2011) used crowdsourcing to acquire multiple independant descriptions of short videos from contributors speaking different languages. We extracted from this Microsoft Video Description Corpus 50 description pairs in French. Note, however, that these descriptions did not have the "verified" status indicating that their contributors were manually selected. In Table 1, we give an example of two descriptions in French of a same short video, with one corresponding description in English (in italics) from the corpus.

Clustered news headlines
News article headlines can be regarded as a potentially rich source of paraphrases, as a single event may be reported in diverse ways (Wubben et al., 2009). News headlines can be easily collected from the Web, for example by using a news aggregation service. We used headlines from the French version of Google News 2 clusters: we initially retrieved 100 clusters from the service, and kept the 50 headline pairs with minimal edit cost as given by TER (Snover et al., 2006). Table 1 gives an example of such a pair, with a possible translation into English.

Web-based paraphrasing game
Paraphrase acquisition has also been formulated as a language game (Chklovski, 2005). We have designed and implemented a Web-based game for sub-sentential paraphrase acquisition, where players have to propose paraphrases in some predetermined context. The rules of the game give bonus points to players who propose paraphrases that receive high marks from other players (who do not take part in the same games) and are rare relative to the set of proposed paraphrases. We have collected paraphrases from 22 French-speaking volunteers who contributed paraphrases for 20 sentential paraphrase pairs. Half of them were given the first sentence of a sentential paraphrase pair for context, and had to paraphrase the appropriate phrase, while the other half were given the second sentence and the corresponding phrase to paraphrase. An example is given in the last row of Table 1.

Annotations and results
We now analyze the results of an annotation experiments on French (section 4.1.) and detail a typology of sub-sentential paraphrases that can be acquired from all corpora under study (section 4.2.).
• Several speakers considered this to be long overdue. MULTIPLE LANGUAGE TRANSLATION → Plusieurs intervenants l'ont considéré comme une chose indispensable.
• Several speakers considered this to be long overdue.   Table 2 provides various statistics for the corpora that were built. The first observation is that MULTIPLE LANGUAGE TRANSLATION and SINGLE LANGUAGE TRANSLATION contain significantly larger sentences (resp. 25.5 and 21.4 tokens on average), almost twice longer than those of SUB-TITLES. VIDEO DESCRIPTIONS contain very short sentences. The case of PARAPHRASING GAME is different as we only take into account the phrases which have been paraphrased, so we only report the number of tokens for those phrases. To compare the similarity of the sentence pairs, we compute different scores, including various metrics designed to evaluate machine translation output:

Annotation experiment and analysis
• an Overlap Coefficient (OC), which represents the percentage of lexical overlap between the vocabularies of the sentence pairs (computed on lemmas); • the BLEU score (Papineni et al., 2002), based on ngram precision; • the TER score (Snover et al., 2006) based on edit rate; • the METEOR score (Lavie and Agarwal, 2007), based on an harmonic mean of precision and recall 3 .
SUBTITLES is the type of corpus with the lowest similarity values. MULTIPLE LANGUAGE TRANSLATION produces, not surprinsigly, paraphrases that are less similar than those produced by SINGLE LANGUAGE TRANSLATIONS. Overall, this latter corpus type contains the most similar paraphrase pairs.
We also conducted experiments to study the coverage of gold standard sub-sentential paraphrases. To this end, we set up annotation work to annotate the sub-sentential paraphrases in the sentence pairs of the different corpora. Annotation was performed by following most of the guidelines from (Cohn et al., 2008) 4 using the YAWAT Web-based tool (Germann, 2008), . The main guidelines are that sure and possible paraphrases must be distinguished, smaller alignments are to be prefered but any-to-any alignments may be used, and sentences should be aligned as much as possible. Henceforth, we will only consider for all reported statistics and experiments those paraphrases that are not identity pairs (e.g. (nice umbrella ↔ nice umbrella)), as they may be considered as trivial as far as acquisition is concerned. Table 3 reports inter-annotator agreement 5 values computed on sets of 50 sentence pairs. We observe that acceptable values are obtained for sure paraphrases, but that low values are obtained for possible paraphrases. This was somehow expected, given the many possible interpretations of possible paraphrases.  Table 2: Statistics on all our corpora (50 sentences) with sentence pair average similarities using several metrics: Overlap Coefficient, TER, BLEU and METEOR the two TRANSLATION corpora contain roughly the same number of tokens in paraphrases (1,234 vs. 1,220), but that the proportion of sure and possible is inversed: the paraphrases found in translations from the same language appear to be much more certain to our annotators than those obtained from multiple languages, which is certainly a result of translation sequences. SINGLE LANGUAGE TRANS-LATION contains significantly more paraphrases than the other corpora, followed by the SUBTITLES corpus. At the other extreme, the VIDEO DESCRIPTION corpus contains significantly fewer paraphrases, both in proportion and number. While the translation-based methods facilitate the identification of paraphrases, especially when they are obtained from a single language, such corpora are of limited availability since multiple translations on a large scale are not available. Furthermore, there are not available for any domain, and their construction is time-consuming and requires skilled translators. News headlines are on the other hand easy to acquire, but they are from a limited genre, which implies that extracted paraphrases could be of limited applicability on other genres and domains.

4.2.
Typology of sub-sentential paraphrases per corpus type After the quantitative study of section 4.1., we now consider the nature of the obtained sub-sentential paraphrases. We first define different types of paraphrases, and compare the results obtained for all our corpora. The different classes that we have asked our annotators to annotate were: • agreement variations: this class includes number and gender variations for nominal or adjectival phrases, and tense modifications of verbs, e.g. souhaites ↔ souhaite (wish ↔ wishes); • inclusion: cases where one of the phrases is more precise than the other, e.g. droits qu'ils ont acquis ↔ droits acquis (rights that they acquired ↔ acquired rights); • typographical variations: includes all variations in number (letters or digits), as well as the use of acronyms vs. complete names, e.g. UE ↔ Union Européenne (EU ↔ European Union); • morphological variations: e.g. en Chine ↔ chinois (in China ↔ chinese); • syntactic variations: where the paraphrases correspond to different syntactic construction, e.g. Souvenons-nous ↔ On se rappelle (Let us recall ↔ We remember); • synonymy: different cases of lexical and phrasal equivalence, e.g. Vous allez adorer ça ↔ ça va vous plaire (you are going to love it); • pragmatic variations: variations which may only be recognized as paraphrases in the context in which they are encountered, e.g. manifestants ↔ écologistes (protesters ↔ green activists), au Bangladesh ↔ dans ce pays (in Bangladesh ↔ in this country). Table 4 reports the results we obtained. We observe that Synonymy (lexical or phrasal) is the most represented category for all corpus type. However, this category only represents 28.3% for NEWS HEADLINES, significantly less than in the other corpora. We further see that the NEWS HEADLINES corpus contains balanced proportions of all types of paraphrases, while, for instance, the Synonymy and Agreement categories account for three quarters of all paraphrases in the SIMPLE LANGUAGE TRANSLATION corpus. 6 Comparing the two translated corpora, we observe that the main difference is due to the number of inclusion which is more important for translations from multiple languages. The GAME corpus is quite different from the other corpora. Players, by following the rules of the game, proposed mainly synonymy or syntactic constructions, and, for instance, did not considered typographical variations as interesting. If this corpus contais a similar proportion of synonymy than the other corpora, we note that it contains significantly more syntactic variations than any other corpus.

Conclusions
In this article, we have proposed a contrastive review of the types and numbers of sub-sentential paraphrases that can be found in various types of sentence pair corpora in French.
We have observed that the most parallel sentence pairs in our study were obtained with SINGLE LANGUAGE TRANS-LATION, and that the most dissimilar are from VIDEO DE-SCRIPTIONS. Not surprisingly, we have found that the most similar sentence pairs contain the most sub-sentential paraphrases, especially sure paraphrases. However, this type of  Table 4: Distribution of paraphrase categories in 50 randomly selected sentence pairs from each corpus corpus is obviously difficult to obtain or costly to build. On the other hand, MULTIPLY-TRANSLATED SUBTITLES and NEWS HEADLINES can be easily collected, on a daily basis for the latter type. These corpora contain sentence pairs that are quite similar, and contain interesting numbers of subsentential paraphrases, even if they contain slightly more possible than sure paraphrases. We have also noted that the NEWS HEADLINES corpus type contains paraphrases of all types. Lastly, we have shown that a corpus from a PARAPHRASING GAME could be used to obtain useful subsentential paraphrases. Figure 1 provides a subjective account of the characteristics of all the studied corpus types which may be useful to select an appropriate corpus for paraphrasing studies or paraphrase acquisition. The characteristics that were considered are the following: