{'fraction_non_alphanumeric': 0.02796516619649209, 'fraction_numerical': 0.008708450876977799, 'mean_word_length': 4.949288580809923, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MTIL17: English to Indian Langauge Statistical Machine Translation

English to Indian language machine translation poses the challenge of structural and morphological divergence. This paper describes English to Indian language statistical machine translation using pre-ordering and suffix separation. The pre-ordering uses rules to transfer the structure of the source sentences prior to training and translation. This syntactic restructuring helps statistical machine translation to tackle the structural divergence and hence better translation quality. The suffix separation is used to tackle the morphological divergence between English and highly agglutinative Indian languages. We demonstrate that the use of pre-ordering and suffix separation helps in improving the quality of English to Indian Language machine translation.


Introduction
In this paper, we present our Statistical Machine Translation (SMT) experiments from English to Tamil, Malayalam, Punjabi and Hindi. From the set of target languages involved, Hindi and Punjabi belong to the Indo-Aryan language family and Tamil and Malayalam belong to the Dravidian language family. All languages except English, have the same flexibility towards word order, canonically following the Subject-Object-Verb (SOV) structure, whereas English follows the Subject-Verb-Object (SVO) structure.
English The structural difference between source and target languages makes SMT difficult. It has been demonstrated that pre-ordering benefits SMT in such cases [32,33]. Pre-ordering or reordering transforms the source sentence into a target-like order using syntactic parse of the source side. After reordering, training of the SMT system is performed using parallel corpus. Reordering also applies to the new source sentences prior decoding. The reordering system is generally developed using rich set of rules for the structural transformation of English sentence into SOV structure. These rules are manually extracted based on analysis 2 R. N. Patel, P. B. Pimpale and M. Sasikumar of source sentence tree and target translations. We have made use of a system developed by [23] to achieve the pre-ordering.
With reference to the morphology, Tamil and Malayalam are more agglutinative compared to English. It is also known that SMT produces more unknown words resulting in bad translation quality, if morphological divergence between source and target language is high. [14,26,30] and [25] have demonstrated ways to handle this issue with morphological segmentation of words before training the SMT system. To tackle the morphological divergence of English with Tamil we have used suffix separation developed by [24,25] as a pre-processing step. The suffix separation tries to reduce the morphological divergence between source and agglutinative target language by splitting long words. Such words when generated in decoding are combined to form a single word in post-processing.
A factored SMT with stem as an alignment factor [12] has been used to achieve better alignment. The target side transliteration is also applied to non translated words.
The rest of the paper is organized as follows. In section 2, we discuss the challenges of english to Indian language machine translation. Section 3 describes the dataset and experimental setup. Section 4 discusses experiments and results followed by a description of our submission to the shared task in section 5. In section 6, we report few early observations and conclusion and future work in section 7.

Challenges of English to Indian Language SMT
As discussed briefly in the introduction, English to Indian language MT poses challenges of structural and morphological differences. In the subsection we discuss the difference of word order and morphology with examples.

Word Order
The important structural difference in English and most of the Indian languages is that of word order. English uses the Subject-Verb-Object (SVO) order and most of the Indian languages, including the ones under study, primarily use Subject-Object-Verb (SOV). Sometimes, these languages exhibit free word order also. Prepositions in case of english come after the pronoun or noun they qualify and for hindi they succeed the noun or pronouns.

Morphology -Agglutination
We discuss here morphological divergence of Tamil and Malayalam with respect to English using analysis of the parallel corpus detailed in table 2. We know that the parallel corpus represents the same information in two different languages. In table 2, we can see that english makes use of more words to represent the same concept or information as compared to tamil and malayalam. If we look at the unique words for each language, we can conclude that english has much less vocabulary as compared to these two Indian languages and so english needs to make use of different combinations of available words to represent a concept. Whereas in case of tamil and malayalam there exist words to represent the same concepts. Examples of the same can be seen in table 3. The different average sentence lengths of these languages also establish the same fact. The significant difference in 2 average word length shows that the words of tamil and malayalam are longer as compared to that of the english. Many words in these Indian languages are formed using compounding of multiple words. The phenomenon is called agglutination and so we say that tamil and malayalam are more agglutinative compared to english.
The difference in length of source and target sentences makes the word alignment difficult. The difficulty ultimately results into poor quality translation system. In our experiments we try to tackle the same by using suffix separation methods for english-tamil SMT.  have to go pokanuma that too aTavuma Table 3. Example english phrases and equivalent tamil words

System Setup
In the following subsections we describe Data distribution followed by pre-processing, evaluation metrics, and SMT system setup for the experiments.

Data set
For our experiments, we have used corpus shared by MTIL-2017 detailed in Table  4. We split the shared data into train, test, and development sets. We used publicly available 3 Indian Language tokenizer and text normalizer for all the target languages. For English, we used tokenizer available with 4 moses . Also, we removed the sentences having word count > 80 or source-target length ratio > 1:9. The details of other advanced preprocessing stages are described in the following subsections.

Preprocessing
To handle the structural divergence between English and other languages, we have used source side reordering. Reordering is proven to improve the translation quality of the language pairs with high structural divergence. To tackle the morphological divergence between the source and target languages for the purpose of a better SMT system, we preprocessed the Tamil for suffix separation. A detailed description of both reordering and suffix separation is given in the following paragraphs.
Reordering (RO) is a preprocessing stage for Statistical Machine Translation (SMT) system where the words of the source sentence are reordered as per the syntax of the target language. The idea is to facilitate the training process by better alignments and parallel phrase extraction for a phrase-based SMT system. Reordering also helps the decoding process and hence improving the machine translation quality.
For English-Hindi SMT, earlier reordering is used by [23,32,33] and have shown significant improvements over baseline. [15] reported SMT results for English to 5 10 major Indian languages and showed that reordering helped for all of them.
Other language pairs have also shown significant improvement when reordering is employed. [37] and [36] have observed improvement for French-English and Chinese-English language pairs respectively. [20] have proposed sentence restructuring whereas [4] have proposed clause restructuring to improve German-English SMT. [27,28] have also reported the use of simple local transformation rules for Spanish-English and Serbian-English translation. Recently, [11] proposed a reordering technique using a deterministic approach for long distance reordering and non-deterministic approach for short distance reordering exploiting morphological information. Some reordering approaches are also presented exploiting the SMT itself [5,9].
Suffix Separation (SS) is the process where the words are split into stem and suffixes. For machine translation, the splitting of an unknown word into its parts enables the translation of the word by the translation of its parts. For example (Hindi-Marathi SMT), in Marathi, 'mahinyaaMnii' is translated as 'mahiine meM' (in the month) in Hindi. In this case, we split the word into 'mahiny + aaMnii'. Here, the suffix 'aaMnii' corresponds to the word 'meM' in Hindi.
We considered only suffix from target language which corresponds to preposition in the source language (English). For this task, the list of suffixes is manually created with the linguistic expertise. When a word is subjected to SS, longest matching suffix from the list is considered for suffix separation. Suffix separation takes place only once for a word. We add a continuation symbol "@@" after the stem word (mahiny@@), which is used to combine the suffixes back after translation. Pseudocode for the suffix separation is detailed in Algorithms 1. Many researchers have tried compound word splitting and suffix separation for SMT between morphologically rich languages. [2] has proposed an approach guided by a parallel corpus. The work is limited to breaking compounds into cognates and words found in a translation lexicon, but no results on translation performance are reported. [13] have demonstrated an empirical method of learning the compound splitting using monolingual and bilingual data and reported impact on performance of SMT. [24,25] reported significantly improved translation quality for Indian languages SMT using suffix separation and compound word splitting.

Transliteration
Out-of-Vocabulary (OOV) words occur in almost all Machine Translation (MT) systems. These words are mostly named entities, technical terms or foreign words that were not part of training corpus or were not added to the development dictionary. So, OOV words need to be translated to the target language using transliteration. Transliteration helps to improve the translation quality [24] and it has also been shown to be useful for translating closely related language pairs [6,19]. For 7 most of the language pairs parallel corpus of transliterations isn't readily available and even if such a training data is made available, the arrangement to integrate transliterated words into MT pipelines are not available in SMT toolkits like phrasal [8] and joshua [31].
Generally, a transliteration system is trained separately outside of an MT pipeline using supervised training methods. It gives all possible target transliterations for a given source word. Generally, the 1-best output is selected as transliteration and is used to replace the OOV word in the translation, post decoding.
This paper uses unsupervised model [7] based on the Expectation Maximization (EM) to induce transliteration corpus from word aligned parallel data, which is then used to train a transliteration model. The implementation is available with the 6 moses toolkit. We use top n-best transliteration output for OOV words. These candidates are plugged in and re-scored with the language model to get the best translation for source sentence.

SMT system set up
The baseline system was setup by using the phrase-based model [1,14,17,21] and [12] was used for factored model. We tuned the model parameters using minimum error rate training (MERT) [21]. The language model was trained using KenLM [10] toolkit with modified Kneser-Ney smoothing [3].We tried various n-gram language models and found that 5-gram performs best for the languages under study. For factored SMT training source and target side stem has been used as alignment factor. Stemming for Hindi, Punjabi, Tamil, and Malayalam, has been done using a modified version of lightweight stemmer [34]. For English we have used porter stemmer [18].

Evaluation metrics
The different experimental systems are being compared using, BLEU [22], PER [29], TER [35], and CDER [16]. For an MT system to be better, higher BLEU and PER scores with lower TER, and CDER are desired.

Experiments and Results
We carried out various experiments to achieve better accuracy, using the data and setup described in previous sections. Table 5 details systems we tried. We also report BLEU, 1-TER, PER, and 1-CDER scores of those systems. It can be seen that the use of preprocessing and transliteration has contributed to the improvement of 1 to 1.5 BLEU points over the baseline for english-hindi, english-punjabi and english-tamil. For english-malayalam the BLEU has decreased and we plan to investigate the same in future. Also, investigation is needed to figure out why the BLEU score decreased on use of factors in english-punjabi, while it was useful for other language pairs. Table 6 describes with an example how reordering reduces the structural divergence and helps to achieve better translation quality. From the example, it can be seen that the translation of the system using S3 is better than the S1. The output of S3 is structurally more correct and conveys the same meaning as that of the reference translation. input Ahmedabad was named after the sultan Ahmed Shah, who built the city in 1411. (english) S1 ahmedabad was named after the sultan ahmed shah, who built the city in 1411.

Error Analysis
A closer look at the performance of these systems to understand the utility of Reordering and Suffix Separation has been done. We report a few early observations.

Reordering Errors
We have used Reodering system developed by [23]. It's based on manual observation and needs the addition of multiple rules. Table 8 details an example of the reordering error. In the example, the phrase sequence 'very useful for losing fat' is wrongly reordered and that has resulted into a wrong translation. The wrong reordering not just affects the structure of the output, but also badly affects the phrase translation.

English
Certain foods are very useful for losing fat.

Reordered
Certain foods very useful fat losing for are .
System Output kuCha khaadya padaarthoM ko khone ke lie bahuta upayogii phaiTa hote haiM Expected Reordering Certain foods fats losing for very useful are .

Bad Split
The suffix separation by [25] is used. For tamil it has limited list of manually observed suffixes and hence it doesn't work for many words. As suffixes are crudely chopped without consideration of validity of remaining part, the errors get introduced. Most of the errors belong to the category where the words which are not meant to split, get split because they end with a certain suffix. This causes sparsity of these genuine terms in the data and leads to a wrong translation of those. For example, a genuine word, say 'abcd' is getting split into 'ab' + 'cd' which is a wrong split. As, 'abcd' is a proper noun and hence should not have been split. Avoiding suffix separation of words in the NNP POS tag was tried, but that was stopping many other valid candidates from pre-processing. A word getting split at wrong position was also one of the major error cases. For example, a word with character sequence 'pqrstu' was getting split into 'pqr' and 'stu' instead of 'pqrs' and 'tu'. In such cases, both suffixes 'stu' and 'tu' are valid and so deciding on when it goes wrong is difficult.

Conclusion and Future Work
In this paper, we presented various systems for translation from English to Hindi, Malayalam, Punjabi, and Tamil. Factored SMT with suffix separation and reordering as pre-processing and transliteration as post-processing performs better. In the immediate future we plan to investigate the detailed impact of the methods devised. For example, failure of factored SMT for english-punjabi and english-malayalam and impact of methods on error rates. Further, we plan to work towards improving the preprocessing and post-processing techniques for better translation quality and extend the approach to other Indian languages.