{'fraction_non_alphanumeric': 0.028881222276983736, 'fraction_numerical': 0.0068013799901429275, 'mean_word_length': 5.120965309200603, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Enlarging Scarce In-domain English-Croatian Corpus for SMT of MOOCs Using Serbian

Massive Open Online Courses have been growing rapidly in size and impact. Yet the language barrier constitutes a major growth impediment in reaching out all people and educating all citizens. A vast majority of educational material is available only in English, and state-of-the-art machine translation systems still have not been tailored for this peculiar genre. In addition, a mere collection of appropriate in-domain training material is a challenging task. In this work, we investigate statistical machine translation of lecture subtitles from English into Croatian, which is morphologically rich and generally weakly supported, especially for the educational domain. We show that results comparable with publicly available systems trained on much larger data can be achieved if a small in-domain training set is used in combination with additional in-domain corpus originating from the closely related Serbian language.


Introduction
Massive Open Online Courses (MOOCs) have been growing rapidly in size and importance, but the language barrier constitutes a major obstacle in reaching out all people and educating all citizens. A vast majority of materials is available only in English, and state-of-the-art machine translation (MT) systems still have not been tailored for this type of texts: the specific type of spoken language used in lectures, ungrammatical and/or incomplete segments in subtitles, slides and assignments, a number of distinct courses i.e. domains such as various natural sciences, computer science, engineering, philosophy, history, music, etc.
Machine translation of this genre into an under-resourced morphologically rich target language represents an additional challenge -in this work, we investigate translation into Croatian. Croatian has recently become the third official South Slavic language in the EU, 1 but it is still rather under-resourced in terms of free/open-source language resources and tools, especially in terms of parallel bilingual corpora. Finding appropriate parallel educational data is even more difficult. Therefore, we based our experiments on a small in-domain parallel corpus containing about 12k parallel segments. We then investigate in what way the translation quality can be improved by an additional in-domain corpus of about 50k segments containing a closely related language, namely Serbian. In addition, we explore the impact of adding a relatively large (200k) out-of-domain news corpus.
Croatian and Serbian are rather close languages, so one option could be to directly use additional English-Serbian data. However, previous work has shown a significant drop in translation quality for a similar cross-language translation scenario (Popović and Ljubešić, 2014). Therefore we also investigate a high-quality Serbian-to-Croatian rule-based MT system for creating additional artificial English-Croatian data.

Related work
In the last decade, several SMT systems have been built for various South Slavic languages and English. Through the transLectures project, 2 transcriptions and translation technologies for automatic generation of multilingual subtitles for online educational data were provided for a set of language pairs. Through this project, one of the South Slavic languages, namely Slovenian, became an optional language pair in the 2013 evaluation campaign of IWSLT (International Workshop on Spoken Language Translation) (Cettolo et al., 2013). The SUMAT project 3 included translation between Serbian and Slovenian subtitles (Etchegoyhen et al., 2014), however translation from English was not explored and no educational subtitles were used.
For Croatian-English language pair, first results are reported in (Ljubešić et al., 2010) on a small weather forecast corpus. Translation between Croatian and English has become one of the focuses of the AbuMatran project: 4 an SMT system for the tourist domain is presented in (Toral et al., 2014), the use of morpho-syntactic information by means of factored models is investigated in (Sánchez-Cartagena et al., 2016) and several scenarios with different models and data-sets are explored in (Toral et al., 2016). Furthermore, SMT systems for the news domain for Croatian and Serbian are described in (Popović and Ljubešić, 2014).
To the best of our knowledge, no systematic investigation on English-to-Croatian educational data has been carried out yet.

Challenges for machine translation of MOOCs
As already mentioned, machine translation of MOOCs induces several challenging tasks. A first step for building a statistical machine translation (SMT) is collection of parallel data. For educational genre, already this step is a challenge due to the following reasons: • crawling: the structure of the web resource containing desired material is complex and does not allow for large-scale automatic crawling; • data extraction and alignment: a large portion of materials is in pdf format, which can lead to misalignments during conversion into plain text; • the size of extracted data: the available data is often very small, so that machine translation with scarce resources has to take place; • target languages: majority of materials is available in English, meaning that machine translation into morphologically rich languages has to take place; • representativeness: majority of available materials are lecture subtitles; slides, notes and assignments are unfortunately rarely translated.
• copyright issues are often not clear and are difficult to define.
Once the parallel data is extracted from some source, segmentation can be a challenging task itself. The platforms used for translation are primarily designed for subtitles so that the translators are encouraged to use short segments which often represent incomplete and/or ungrammatical sentences. Another peculiarity is the fact that the lecturers often do not finish a sentence properly or change the subject in the middle of a utterance.

Challenges for English-Croatian machine translation and getting help from Serbian
Croatian, as a Slavic language, has a very rich inflectional morphology for all word classes. There are six distinct cases, three genders and a number of verb inflections since person and many tenses are expressed by the suffix. In addition, negation of three important verbs is formed by adding the negative particle to the verb as a prefix. As for syntax, the language has a quite free word order, and there are no articles, neither indefinite nor definite. In addition, multiple negation is always used. All these morpho-syntactic peculiarities are even more difficult to generate correctly if the available resources are scarce, especially for spoken language style used in lectures as well as for ungrammatical and/or unfinished sentences.
Differences between Croatian and Serbian Both languages belong to the South-Western Slavic branch. Although they exibit a large overlap in vocabulary and a strong morpho-syntactic similarity so that the speakers can understand each other without difficulties, there is a number of small but notable and frequently occurring differences between them.
The largest differences between the two languages are in vocabulary: some words are completely different, some however differ only by one or two letters. In addition, Serbian language usually phonetically transcribes foreign names and words although both transcription and transliteration are allowed, whereas the Croatian standard only transliterates.
Apart from lexical differences, there are also structural differences, mainly concerning verbs: constructions involving modal verbs, especially those with the verb "trebati" (to need, should), future tense, conditional.

Research questions
Taking into account the facts described in previous sections, i.e. peculiarities of educational genre, difficulties regarding characteristics of the Croatian language and scarceness of available resources, as well as similarities and differences between Croatian and Serbian, our main questions are: • how does the translation performance for a small in-domain training data compare with the performance for a larger out-of-domain data?
• is it possible to increase the performance by adding Serbian in-domain data and what is the optimal way?
Our work is to certain extent related to the experiments described in (Popović and Ljubešić, 2014). They explored adaptation of the in-domain news test data to the desired language by applying a set of simple rules or a Serbian-Croatian SMT system, whereas the training data for the English-Serbian/Croatian system were fixed. We investigate different combinations of training data for the challenging genre i.e. educational material in order to build a ready SMT so that the test data once translated do not require any further intervention. In addition, we use a recently developed high quality rule-based Serbian-to-Croatian system  which performs better than SMT systems used in (Popović and Ljubešić, 2014).

Experimental set-up Parallel texts
The data used in our experiments are collaboratively translated subtitles from Coursera 5 and contain several types of courses/domains: biology, computer science, philosophy, nutrition, music, etc. The translations are produced by course participants who usually translate into their native languages. Translations are done via a collaborative platform which is usually used for translation of movie subtitles, thus not designed with large-scale crawling in mind. In order to crawl the relevant data, we first had to construct manually a list of the Coursera courses available there. Once the list of translated Coursera courses was constructed, Python scripts were used to download the original English data and the corresponding translations. However, this process was not fully automatic because there were some issues with the format of the URLs of some of the courses as well as the data format of the translations.
The parallel data collected is of a relatively good quality. The texts are mostly properly aligned, however the sentence segmentation is not optimal. As mentioned in Section 2, the extracted parallel segments often contain incomplete sentences or parts of two different sentences. Of course, one can think of automated correction of segmentation. However, for bilingually aligned texts this represents a peculiar task for several reasons: • there are no apparent punctuation rules which are consistent in both languages: some sentences end with "." i one language but with "," or ";" or conjunction or even nothing in the other; • some consecutive English source segments are only partially translated (Table 1) -if these segments were merged in both languages, a proper English sentence aligned with an incorrect and/or ungrammatical translation would be generated.

English Croatian Five years ago
Pre pet godina I was told specifically (no translation) this is his name da mu je to ime. For these reasons, and also taking into account the fact that the test set is in the same format, no resegmentation attempts were performed and the texts are used directly in the format they were extracted. Nevertheless, since the data are not completely clean, certain filtering steps had to be performed. First of all, there was a large number of short redundant segments such as "Mhm", "Hello", "Welcome back", etc. These segments were separated from the rest according to the sentence length and only a unique occurrences were kept. The rest of the corpus was then cleaned from incorrect translations on the base of sentence length: if the proportion of source and target sentence length was too high or too small, the segment was removed. As a final step, the two cleaned parts of the corpus were merged. The same procedure was carried out for both for English-Croatian as well as for English-Serbian data sets. For English-Croatian, about 12k parallel segments were extracted, and for English-Serbian about 50k. An interesting observation is that although Croatian is generally better supported in terms of publicly available parallel data, 6 Serbian is currently better supported for educational parallel texts.
As for the out-of-domain corpus, we used the SETimes news corpus (Tyers and Alperen, 2010) since it is relatively large (200k parallel sentences) and clean.

Moses set-ups
We trained the statistical phrase-based systems using the Moses toolkit (Koehn et al., 2007) with MERT tuning. The word alignments were built with GIZA++ (Och and Ney, 2003) and a 5-gram language model was built with SRILM (Stolcke, 2002).
The investigated bilingual training set-ups are: where hr' denotes Serbian part of the corpus translated by a rule-based machine translation system into Croatian. For each set-up, the language model was trained on the target part of the used bilingual corpus. For set-ups including combined parallel corpora (3, 4 and 5), the corpora were merged by simple concatenation and the interpolated language model was used. Data statistics for all set-ups can be seen in Table 2.
Serbian-to-Croatian RBMT system The MT system  used for creating additional artificial Croatian data from Serbian is a bidirectional rule-based system based on the open-source Apertium platform (Forcada et al., 2011). Considering the fact that differences between Croatian and Serbian occur mostly at the lexical and orthography, using a rule-based system makes the most sense. The system tested on newspaper texts achieves 83.0% BLEU for translation into Croatian, whereas the BLEU score is 72.7% if the Serbian source is directly compared to the Croatian reference translation.

Evaluation
For all set-ups, BLEU scores (Papineni et al., 2002) and character n-gram F-scores i.e. CHRF3 scores (Popović, 2015) are reported. In addition, five Hjerson error classes (Popović, 2011) are reported in order to get a better insight into differences between the systems: inflectional errors, ordering errors, missing words, additions and lexical errors. Table 3 presents the obtained automatic scores for all Moses training set-ups described in Section 5 together with the scores for translations generated 7 by two publicly available SMT systems for Englishto-Croatian: Asistent 8 (Arčan et al., 2016) and Google translate 9 . It can be seen that the most promising set-up according to automatic evaluation metrics is the set-up 5, i.e merging both domains and adding artificial in-domain English-Croatian parallel text where the target Croatian part is generated from Serbian by the rule-based MT system. This set-up even outperforms the Asistent system which is trained on much larger parallel texts, albeit none of them from educational domain. Furthermore, it can be seen that both SETimes and original Coursera set produce the same percentage of lexical errors -the first one due to the domain discrepance and the other due to data sparsity. Adding in-domain Serbian data reduces the number of lexical errors, which is further reduced by translating Serbian into Croatian. Merging of two data-sets reduces lexical errors even more, however their number is still larger than for Asistent and Google systems.  Table 3: Automatic evaluation scores (%) for each of the systems: BLEU score, CHRF3 score and five Hjerson error rates: inflectional, ordering, omission, addition and lexical error rate together with their sum.

Automatic evaluation scores
Ordering errors and omissions are lower for the set-ups without SEtimes, most probably due to different sentence (i.e. segment) structure in two genres/domains.
Morphological errors are also lower without SEtimes, however they are high in all set-ups which should be generally addressed in future work by using morpho-syntactic analysers and/or generators.
Apart from this, it can be observed that the main advantage of the Google system is the low number of lexical errors which is probably achieved by using very large training corpora.

Translation examples
In order to illustrate advantages and disadvantages of different SMT systems, Table 4 shows six English source segments and their translations by each of the systems. Erroneous parts of the obtained translations are annotated by parentheses: {} stands for lexical errors, additions, omissions and inflections (where only part of the word is in parenthesis), // stands for ordering errors and <> for stylistic variants. segment 1: A completely correct sentence is produced only by the set-up 5, as well as by the publicly available systems. The other systems generate ungrammatical sentences, en-hr Coursera alone generates stylistically questionable translation.
segment 2: None of the systems produces a perfect translation -however, the most accurate translation containing only two minor morphological errors is produced by set-up 5, i.e. combination of all Coursera data and SETimes.
segment 3: Spoken lecture language issues: SEtimes produces the worst translation, followed by Google and then Asistent; all set-ups with Coursera data produce correct translations.
segment 4: The translation of the incomplete segment is difficult for all systems. Both SETimes and Croatian Coursera alone generate very bad translations -the first one because of domain discrepancy, the second one because of data sparsity; the other set-ups generate ungrammatical segments where the meaning still can be captured. Asistent produces the best translation containing only one inflectional error which does not change the meaning.
segment 5: The best translation (without any error) of another incomplete sentence is generated by setup 4, i.e. Coursera with additional artificial data; the worst translation is generated by SEtimes, which also introduces morphological errors when combined with Coursera in set-up 5. This example illustrates that in-domain data are important not only for vocabulary and lexical errors but also for morpho-syntactic properties.
segment 6: Spoken language and incomplete sentence: SETimes, Google and Asistent produce a number of errors; using Serbian instead of Croatian induces some errors mainly due to differences in verb structures; the best option is the use of Croatian Coursera with or without additional data.  It can be noted that the additional Serbian data does not help only in the first example -only when a larger out-of-domain data is added, a correct translation is obtained. For segment 2), the baseline English-Croatian corpus already yielded a correct translation and there is no change when any of additional corpora is used. In example 3) both the direct use of Serbian and translating into Croatian help to some extent but some errors are still present. For segments 4) and 5) both untranslated and translated Serbian texts result in the same correct translation. For the example 6) using the translated additional data significantly improves the performance in comparison with the "raw" Serbian data.

Summary and outlook
This work has shown that a small amount of in-domain training data is very important for the English-to-Croatian statistical machine translation of the specific genre of Massive Open Online Courses, especially for capturing appropriate morpho-syntactic structure. Adding in-domain data containing the closely related Serbian language improves the performance, especially when the Serbian part is translated into Croatian thus producing an artificial English-Croatian in-domain corpus. The improvements consist mainly from reducing the number of lexical errors. Further improvements have been achieved by adding a relatively large out-of-domain news corpus reaching performance comparable with systems trained on much larger (out-of-domain) parallel texts. Adding this corpus reduces the number of additions and lexical errors, nevertheless it introduces more morphological and ordering errors due to the different nature and structure of the segments.
Future work should include investigating better ways of combining and extracting relevant information from original (in-domain) and additional (out-of-domain and/or "out-of-language") data. In addition, the use of morpho-syntactic information should be explored, especially since this also represents a challenging task for the peculiar genre such as educational material.