{'fraction_non_alphanumeric': 0.027301481604794408, 'fraction_numerical': 0.006991842850008324, 'mean_word_length': 5.286013254272759, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PE2rr Corpus: Manual Error Annotation of Automatically Pre-annotated MT Post-edits

We present a freely available corpus containing source language texts from different domains along with their automatically generated translations into several distinct morphologically rich languages, their post-edited versions, and error annotations of the performed post-edit operations. We believe that the corpus will be useful for many different applications. The main advantage of the approach used for creation of the corpus is the fusion of post-editing and error classification tasks, which have usually been seen as two independent tasks, although naturally they are not. We also show benefits of coupling automatic and manual error classification which facilitates the complex manual error annotation task as well as the development of automatic error classification tools. In addition, the approach facilitates annotation of language pair related issues.


Introduction
The amount of post-edited translation outputs as well as the amount of data containing annotated machine translation errors is growing in the recent years, with more and more applications and growing need for such data. The process of improving a machine-generated translation is a natural task -post-edited translations are a by-product of the professional translation workflow, whereas (explicit) translation error classification, which is a very difficult task, is not. On the other hand, the two tasks are actually highly related -post-editing can be viewed as implicit error annotation, since each edit operation is actually a correction of a translation error. Despite of this fact, these two tasks have almost always been seen and performed completely separated and independent. Therefore, data containing error annotations of actual post-edit operations are scarce. The corpus PE 2 rr aims to improve this situation, containing post-edited translation outputs where each post-edit operation is assigned to a particular error class. In addition, another discrepance in the error classification experiments and corpora can be observed: manual error classification and automatic error classification have always been carried out completely separately from each other. Whereas the results of manual error annotation present a valuable resource for development of automatic error classification tools, this independent approach makes detailed analysis of drawbacks of automatic tools difficult. The PE 2 rr corpus has been created by merging these two tasks together: automatic error classification has been performed as a first step, and the manual error classification has been performed as a correction of these automatically preannotated errors. A very important advantage of this approach is also reducing the effort of the complex and timeconsuming manual error classification process, since correcting pre-annotated errors is easier and faster than looking for the errors and defining them from scratch. We believe that the approach can also improve the inter-annotator (and intra-annotator) agreement, however this aspect will be in-vestigated in future work. The benefits for the community can be manifold: • facilitating improvement and development of automatic error classification and error prediction tools; • further usage as additional post-edited and/or error annotated data for quality estimation and error prediction, automatic-post-editing, etc.; • investigating correlations between automatic evaluation metrics and certain errors; • indicating most prominent problems of the state-ofthe-art SMT systems for the described target languages as well as for a number of related ones.
We also believe that our positive experience of joining postediting, automatic and manual error classification will be further used for more language pairs and by professional translators, and that the community will derive valuable knowledge about the annotation process and translation errors from the corpus we presented.

Related work
Publicly available post-edited data have been used for a while for quality estimation and error prediction WMT tasks 1 , also for automatic post-editing task (Bojar et al., 2015). Parts of the corpora are also error annotated, though only with basic edit distance operations (substitution, deletion, insertion and shift) or binary tags ("ok" for correct words and "bad" for erroneous ones). Another corpus containing same type of edit annotation is the TRACE corpus (Wisniewski et al., 2013) which consists of French-English and English-French post-edited translation outputs annotated with basic edit distance error types. Detailed manual error analysis has been often used in the last decade to determine the most prominent errors for particular task/translation system (e.g. (Vilar et al., 2006) at the beginning, (Lommel et al., 2014) recently), as well as to investigate impacts of different error classes to various aspects of translation quality ( (Kirchhoff et al., 2012;Federico et al., 2014). Nevertheless, none of these error analyses have been carried out on post-edits, only on raw translation outputs. Manual error analysis of Finish post-edits has been carried out in (Koponen, 2013) in order to investigate discrepances between the estimated cognitive effort and actual technical effort, however, to the best of our knowledge, the corpus is not publicly available. The publicly available TARAXÜ corpus  contains both post-edited as well as error annotated translation outputs, whereby the two tasks were carried out completely separately, and even not on the same set of translation outputs. The Terra corpus (Fishel et al., 2012) is a publicly available collection of manually error annotated corpora 2 which has been used for assessment of the automatic classification tools Addicter (Zeman et al., 2011) and Hjerson (Popović, 2011). The corpora were annotated by different research teams independently and the annotation strategies were different: from a free annotation using only the source text and the translation output without taking into account any reference translation, to a flexible reference-based annotation where a reference translation has also been taken into account. Although this corpus has been very valuable for automatisation of error classification, all data sets pose difficulties for detailed and precise assessment of automatic tools and their further development, because (i) standard reference translations differ much more from translation outputs than post-edits, so that automatic error annotation tools tag a large number of actually correct words and (ii) manual and automatic error classification were performed completely independently.

Data sets
The corpus PE 2 rr contains Serbian and Slovenian SMT generated translations from English and German source texts, as well as a part of the TARAXÜ corpus, namely German, Spanish and English SMT generated translations from English, Spanish and German source texts. The Serbian and Slovenian languages, as Slavic languages, have quite free word order and are highly inflected. The derivational morphology is also rich, multiple negation is used, and there are no articles, only determiners. Similarly, German is also a morphologically rich language and a very challenging language for machine translation. Spanish is generally less inflective than the Slavic languages and German, however the number of possible verb inflections is rather high. The following domains/genres were used: news texts from the enhanced version 3 of the SETimes (Tyers and Alperen, 2010) corpus for English→Serbian, Eu-roParl (Koehn, 2005) for English/German→Slovenian and OpenSubtitles 4 for all language pairs. All the corpora are downloaded from the OPUS web site 5 (Tiedemann, 2 http://terra.cl.uzh.ch/terra-corpus-collection.html 3 http://nlp.ffzg.hr/resources/corpora/setimes/ 4 http://www.opensubtitles.org/ 5 http://opus.lingfil.uu.se/ 2012). It should be noted that the OpenSubtitles corpus contains transcriptions and translations of spoken language thus being slightly peculiar for machine translation. From the TARAXÜ corpus, the post-edited part containing WMT News texts is used. Table 1 gives an overview of the total amount of sentences and words used for generation of the corpus PE 2 rr for each domain and translation direction. All translations have been generated using phrase-based Moses (Koehn et al., 2007), where the word alignments were built with GIZA++ (Och and Ney, 2003). The 5-gram language model was built with the SRILM toolkit (Stolcke, 2002). For each translation output, a system was trained on the corresponding in-domain parallel data.

Post-editing and error annotation
For the Serbian and Slovenian MT outputs, both tasks, i.e. post-editing and error annotation, were performed by MT researchers with some experience with human translation. The annotators are native speakers of the target languages and fluent in both source languages. For the TARAXÜ data, post-editing was performed by professional translators and error annotation was performed by MT researchers fluent both in the corresponding source and target languages. The evaluation has been carried out in three steps: first, each machine generated translation has been post-edited. Then, automatic error classification has been applied to assign an error category to each post-edit operation. Finally, manual inspection and correction of these error labels has been carried out. In addition to standard error classification, first steps towards the annotation of the most prominent language(-pair) related issues  has been carried out. The general procedure for generation of the PE 2 rr corpus is shown in Figure 1. Rectangle processes were carried out automatically and ellipse processes manually.

Post-editing
Post-editing has been performed to create a fully fluent and adequate translation which is of the same quality level as a standard human translation. Only a minimal post-editing necessary to achieve an acceptable translation quality (adequate and fluent translation) has been performed, without taking into account potential editors' preferences concerning style, lexical choice, grammatical structure, etc. Table 1 shows number of segments and running words in all source texts and in all generated machine translation outputs together with the number of post-edited words (which is about 30% on average).
Apart from that, a division of performed technical effort into four categories is presented in Table 2. It can be seen that overall, about one third of the segments required medium technical effort, 20% segments required only small intervention, 20% were already acceptable, and about 25% required significant modifications. Edit distances for each segment in the form of WER are a part of the corpus, too.

Error annotation
The task of error annotation consisted of assigning an error class to each post-edit operation and was performed in two stages. The first stage consisted of automatic pre-annotation by Hjerson (Popović, 2011), a tool for automatic error classification which enables categorisation into five error classes: addition, inflectional error (verb tense/person/mood, case, gender, number), lexical error (mistranslation), omission, and word order error. For Slovenian translations, lemmas were generated by Obelix (Grčar et al., 2012). For Serbian translations, "poor man's" version of Hjerson , based on first four letters of words instead of lemmas, was used. For the TARAXÜ target languages, the TreeTagger 6 was used. The second stage consisted of correcting or expanding these automatically generated error classes. Three additional error classes which were observed as recurrent and frequent were introduced: • contraction separated, incorrect or missing (parts of) compounds/contractions: Serbian and Slovenian negation particle+verb contractions, German preposition+article contractions, Slovenian preposition+pronoun contractions, Spanish ar-ticle+preposition and verb+pronoun contractions, as well as different types of compounds in all languages.
• derivation incorrect POS, verb aspect, verb prefix, passive/pastparticiple confusion, possessive adjectives and pronouns • untranslated out-of-vocabulary words in the source language which remained untranslated in the output Apart from that, some errors can be assigned to more than one error class -a number of contractions, derivations, inflections, mistranslations and unknown words can be placed at an incorrect position in a sentence, i.e. they are also reordering errors. This phenomenon is not possible to detect by the state-of-the-art automatic error classification tools so that these multiple tags were assigned during the manual evaluation.

Distribution of manual error classes
Distribution of error classes in the corpus PE 2 rr are presented in Table 3 in the form of raw counts and error rates for each of the error categories. The error rates are obtained by dividing raw counts with total number of running words and therefore are suitable for comparison of different translation outputs.

Annotation of language related issues
Different language combinations exhibit different error distributions in the translation output which often relates to linguistic characteristics of involved languages as well as to divergences between them. Those relations have been investigated in  and the following has been discovered: • there is a number of frequent error patterns, i.e. obstacles (issues) for SMT systems • nature and frequency of many issues depend on language combination and translation direction • some of translation errors depend on domain and text type, mostly differing for written and spoken language.
Manual inspection of source sentences and their corresponding machine translations annotated by Hjerson using the independent reference translations together with their corresponding source sentences has been carried out in these experiments, and the phenomena were only counted, not annotated. In this work, we carried out a preliminary experiment of annotating the part of the PE 2 rr corpus with these issues. The labels were assigned on the segment level and the distributions of the ten most frequent issues for each annotated text are presented in Table 4. Finding the best definition of issues and the best method for annotating is still a part of the ongoing research, but detecting and defining an issue is certainly much easier when inspecting errors in the form of edit operations instead of errors with respect to the independent references.

Comparison between manual and automatic error annotation results
The comparison between Hjerson counts and manual classification counts has been carried out in the same way as in (Popović and Burchardt, 2011), namely: • comparison of error distributions within one translation output as well as across different translation outputs in the form of Spearman correlation coefficients; • precision, recall and confusions between different error classes.  Table 5: Precision and recall for Hjerson error categories for the German→English News part of the PE 2 rr corpus compared with the similar text from the Terra corpus. Label "x" stands for correct words (not edited/no error).
The correlations were already very high in the experiments reported in (Popović and Burchardt, 2011), but using the PE 2 rr corpus resulted in a significant increase in recalls and especially precisions for all error classes. Table 5 presents precision and recall for Hjerson classes with regard to manual classes. The results are reported for the German→English News part of the PE 2 rr corpus and compared with the same text type 7 from the Terra corpus analysed in (Popović and Burchardt, 2011) where independent reference translations were used for automatic error classification and the results are afterwards compared with results of a completely independent manual error analysis. This means that the PE 2 rr corpus, where the reference translations are post-edited translation outputs and a manual error annotation has been carried out on pre-annotated texts, can give much better and more reliable insights into particular flaws of an automatic error classification tool. A detailed analysis of Hjerson performance, being an important direction for future experiments, is however out of the scope of this work. Nevertheless, it should be already noted that Hjerson definitely exhibits a significant number of confusions between lexical errors, omissions and additions -addressing this problem should be one of the first steps for its improvement.

Summary and Outlook
We have presented a freely available corpus 8 containing automatically generated translations into several highly inflective languages, their post-edited versions, and error annotations of the performed post-edit operations. We believe that the corpus will be useful for many distinct applications. The main advantage of the approach used for creation of the corpus is the fusion of post-editing and error classification tasks, which have usually been seen as two independent tasks, although naturally they are not. In addition, we show benefits of coupling automatic and manual error classification which facilitates the complex manual error annotation task as well as the development of automatic error classification tools. Future work on the presented corpus includes adding more language pairs and domains to the collection, further annotation and investigation of the language related obstacles, and further in-depth analysis and improvement of the stateof-the-art methods for automatic translation error analysis.  Table 3: Distribution of error classes: number and percentage of words belonging to each of the eight error classes; counts for multiple error classes are added to each of the corresponding categories by "+n". Percentage is calculated with respect to total number of running words.

Acknowledgements
This publication has emanated from research supported by TRAMOOC project (