{'fraction_non_alphanumeric': 0.03209674486270231, 'fraction_numerical': 0.01209310783778869, 'mean_word_length': 4.938984881209503, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Augmenting Pivot based SMT with word segmentation

This paper is an attempt to bridge two well known performance degraders in SMT, viz., (i) difference in morphological characteristics of the two languages, and (ii) scarcity of parallel corpora. We address these two problems using " word segmen-tation " and through " pivots " on the morphologically complex language. Our case study is Malayalam to Hindi SMT. Malay-alam belongs to the Dravidian family of languages and is heavily agglutinative. Hindi is a representative of the Indo-Aryan language family and is morphologically simpler. We use triangulation as pivoting strategy in combination with morphological pre-processing. We observe that (i) significant improvement in translation quality over direct SMT occurs when a pivot is used in combination with direct SMT, (ii) the more the number of pivots, the better the performance and (iii)word segmenta-tion is a must. We achieved an improvement of 9.4 BLEU points which is over 58% compared to the baseline direct system. Our work paves way for SMT of languages that face resource scarcity and have widely divergent morphological characteristics .


Introduction
Hindi (hin) and Malayalam (mal) are two important languages from Indian sub-continent. Hindi is a language belonging to Indo-Aryan family with 300 million native speakers. Malayalam belongs to the Dravidian language family. It is spoken by over 38 million people.
The task of translation between Hindi and Malayalam proves to be a difficult one. This is due to scarcity of available parallel corpus and high agglutinative nature of Malayalam. Malayalam is a morphologically rich, agglutinative language in which complex words are formed by concatenating morphemes together. For example, "अगर बादल नहीं बरसे तो भी" (if cloud not rain_verb then also) in Hindi (5 words) would translate to "മഴാ െപ ുതിെല ിലും" (rain_noun rain_verb+not+even_if+then_also) in Malayalam (2 words).
In this paper, we present a case of translation from Malayalam to Hindi. Our approach is based on combined use of pivot strategies for Statistical Machine Translation (SMT) and word segmentation techniques. We show that word segmentation of source language as well as pivot language helps to improve the translation quality. Section 2 contains details about relevant work done in the field. Section 3 explains the design of our system in detail. Section 4 describes the experimental setup. Results of the experiments are discussed in Section 5. Section 6 includes concluding remarks on the mal-hin translation task.

Related Work
There is substantial amount on pivot-based SMT. De Gispert and Marino (2006) discuss translation tasks between Catalan and English with the use of Spanish as a pivot language. Pivoting is done using two techniques: pipelining of source-pivot and pivot-target SMT systems and direct translation using a synthesized Catalan-English. In Utiyama and Isahara (2007), the authors propose the use of pivot language through -phrase translation (phrase table creation) and sentence translation. Wu and Wang (2007) compare three pivot strategies viz.
-phrase translation (i.e. triangulation), transfer method and synthetic method. Nakov and Ng (2012) try to exploit the similarity between resource-poor languages and resource-rich languages for the translation task. Dabre et al. (2014) used multiple decoding paths (MDP) to overcome 303 the limitation of small sized corpora. Paul et al. (2013) discusses criteria to be considered for selection of good pivot language. Use of sourceside segmentation as pre-processing technique has been demonstrated by (Kunchukuttan et al., 2014). Goldwater and McClosky (2005) investigates several methods for incorporating morphological information to achieve better translation from Czech to English.
Most of the pivot strategies mentioned above focus on the situation of resource-poor languages where direct translation is either very poor or not available. Our approach, like Dabre et al. (2014), tries to employ pivot strategy to help improve the performance of existing SMT systems. To the best of our knowledge, our work is the first attempt to integrate word segmentation with pivot-based SMT.

Our System
We propose a system which integrates word segmentation with triangulation and combines more than one SMT systems. The required concepts are explained as follows.

Pivoting by Triangulation
Wu and Wang (2007) discuss triangulation as a pivoting strategy. In this method, the source-pivot models and pivot-target models are trained using source(L s )-pivot(L p ) and pivot(L p )-target(L t ) corpora respectively. Using these two models, we induce a source-target model. The two important components to be calculated are -1) phrase translation probability and 2) lexical weight.
The Phrase translation probability is estimated by marginalizing over all possible pivot phrase, along with the assumption that the target phrases are independent of the source phrase given the pivot phrase. The phrase translation probability can be calculated as shown below: The Lexical Weight, according to Koehn et al. (2003), depends on -1) word alignment information a in a phrase pair (s, t) and 2) lexical translation probability w(s|t).
Lexical weight can be modeled using following equation, (2) Wu and Wang (2009) discuss in detail about alignments information and lexical translation probability.

Word segmentation
We use unsupervised word segmentation as preprocessing technique. For this purpose, Morfessor (Virpioja et al., 2013) is used. It performs morphological segmentation of words of a natural language, based solely on raw text data. Morfessor uses probabilistic machine learning methods to do the task of segmentation. The trained models for word segmentation of Indian languages are available to use 1 .

Integrating word segmentation with Triangulation
In our system, we use both phrase table triangulation and word-segmentation. The words in the source and pivot language training corpora are segmented before training the SMT systems. The target language is left unchanged. The phrase tables are then triangulated. This process is shown in Figure

Multiple Decoding Paths
We use the triangulated phrase table to supplement the direct phrase table. In order to integrate these two phrase tables, we use the multiple decoding paths (MDP) feature provided by the Moses decoder. Multiple decoding paths (Koehn and Hoang, 2007) allows us to lookup multiple translation models for hypothesis at decoding time, and the choice of best hypothesis at decoding time based on available evidence. We use MDP to combine one or more pivot-based MT systems with the direct MT system. This constitutes our final decoding system. We preferred this option over offline linear interpolation of phrase tables since the framework can dynamically consider phrases from multiple phrase tables and wouldn't need any hyperparameter tuning.

Experiments
The aim of experiments is to study impact of pivot strategies and word segmentation, separately and together.
Our data split was as follows: 46277 sentences are used for training, 500 sentences are used for tuning and 2000 sentences are used for testing.
For the experiments, we use phrase-based SMT training and 5-gram SRILM language model. Tuning is done using the MERT algorithm. The triangulated MT systems use default distance based reordering while direct systems use wbe-msdbidirectional-fe-allff model

Experimental Setup
We trained various phrase based SMT systems by combining the basic systems mentioned in Section 3. We use a threshold of 0.001 for phrase translation probability to manage size of triangulated phrase table. The performance metric used is BLEU (Papineni et al., 2002). The following are the configurations we experimented with: 1. DIR: MT system trained on direct Source-Target corpus.
3. PIVOT: MT system based on triangulated phrase table of Source-Target using a single Pivot language.
4. PIVOT_Morph: PIVOT system with both Source and Pivot texts segmented.
5. PIVOT_SourceMorph: PIVOT system with only Source text segmented.
6. DIR+PIVOT: MT system based on integration of DIR and PIVOT phrase tables using MDP.
8. DIR+All-PIVOT: MT system based on integration of DIR and all 7 PIVOT systems using MDP.
9. DIR_Morph+All-PIVOT_Morph: MT system based on integration of DIR_Morph and all 7 PIVOT_Morph systems using MDP. Table 1 shows BLEU scores for best performing MT systems for each experiment. Wherever a single pivot is used, it refers to the best supplementing pivot. In all the experiments, Punjabi was observed to be the best pivot. We can see that the pivot-only system gives translation accuracy comparable to the direct system for the best performing pivot language, Punjabi. Punjabi shares a large fraction of its vocabulary with Hindi. In addition, pan is morphologically simpler compared to other Indian languages. Hence, better word alignment can be achieved and during triangulation, more phrase pairs are obtained. Though the PIVOT system does not better the performance of the DIRECT system, the combination of both the system i.e. DIR+PIVOT performs better than the DIRECT system. 305

Type of MT
DIR+PIVOT has relative improvement of 6.8% over DIR system. Addition of all PIVOT systems using multiple decoding paths shows that each pivot helps improve the BLEU score since each PIVOT system provides additional translation options. The all-pivot system i.e. DIR+All-PIVOT shows improvement of about 15% (2.56 BLEU points) over DIR system.
Pre-processing the corpus with word segmentation (DIR_Morph system) results in a substantial improvement of 44% over the DIR system. Use of word segmentation brings about a major improvement in the BLEU score. Since mal is morphologically rich, word segmentation reduces data sparsity, helps learn better alignment models and reduces the number of unknown words in test set.
We compared the use of word segmentation at source as well as pivot with word segmentation on the source language alone. Figure 2 compares these two system for various pivot languages. The additional use of word segmentation at pivot provides a 4% to 18% increase in the BLEU score.
The DIR_Morph+All-PIVOT_Morph system combines all pivot languages as well as source and pivot segmentation. This is the best performing system which achieves an improvement of 9.4 BLEU (58% improvement) over DIR system.

Conclusions
We have shown that pre-processing using word segmentation and augmentation of direct systems with pivot-based systems provides significant advancement in translation quality. This is an attempt to integrate word segmentation with pivotstrategies. We achieved compelling results with our approach. The approach is applicable to any resource-scarce language pair with morphologically rich source side. In future, we will focus on applying our approach to other challenging language pairs. We will also work on MT tasks which involve morphologically rich target language.