{'fraction_non_alphanumeric': 0.03413597733711048, 'fraction_numerical': 0.020254957507082155, 'mean_word_length': 5.038272396835579, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Edinburgh’s Statistical Machine Translation Systems for WMT16

This paper describes the University of Edinburgh’s phrase-based and syntax-based submissions to the shared translation tasks of the ACL 2016 First Conference on Machine Translation (WMT16). We submitted five phrase-based and five syntaxbased systems for the news task, plus one phrase-based system for the biomedical task.


Introduction
Edinburgh's submissions to the WMT 2016 news translation task fall into two distinct groups: neural translation systems and statistical translation systems. In this paper, we describe the statistical systems, which includes a mix of phrase-based and syntax-based approaches. We also include a brief description of our phrase-based submission to the WMT16 biomedical translation task. Our neural systems are described separately in Sennrich et al. (2016a).
In most cases, our statistical systems build on last year's, incorporating recent modelling refinements and adding this year's new training data. For Romanian-a new language this year-we paid particular attention to language-specific processing of diacritics. For English→Czech, we experimented with a string-to-tree system, first using Treex 1 (formerly TectoMT; Popel and Žabokrtský, 2010) to produce Czech dependency parses, then converting them to constituency representation and extracting GHKM rules.
In the next two sections, we describe the phrasebased systems, first describing the core setup in Section 2 and then describing system-specific extensions and experimental results for each individual language pair in Section 3. We describe the 1 http://ufal.mff.cuni.cz/treex core syntax-based setup and experiments in Sections 4 and 5.
2 Phrase-based System Overview

Preprocessing
The training data was preprocessed using scripts from the Moses toolkit. We first normalized the data using the normalize-punctuation.perl script, then performed tokenization (using the -a option), and then truecasing. We did not perform any corpus filtering other than the standard Moses method, which removes sentence pairs with extreme length ratios, and sentences longer than 80 tokens.

Word Alignment
For word alignment we used fast_align (Dyer et al., 2013)-except for German↔English, where we used MGIZA++ (Gao and Vogel, 2008)-followed by the standard grow-diag-final-and symmetrization heuristic.

Language Models
Our default approach to language modelling was to train individual models on each monolingual corpus (except CommonCrawl) and then linearlyinterpolate them to produce a single model. For some systems, we added separate neural or Com-monCrawl LMs. Here we outline the various approaches and then in Section 3 we describe the combination used for each language pair.
Interpolated LMs For individual monolingual corpora, we first used lmplz (Heafield et al., 2013) to train count-based 5-gram language models with modified Kneser-Ney smoothing (Chen and Goodman, 1998). We then used the SRILM toolkit (Stolcke, 2002) to linearly interpolate the models using weights tuned to minimize perplexity on the development set.
CommonCrawl LMs Our CommonCrawl language models were trained in the same way as the individual corpus-specific standard models, but were not linearly-interpolated with other LMs. Instead, the log probabilities of CommonCrawl LMs were added as separate features of the systems' linear models.
Neural LMs For some of our phrase-based systems we experimented with feed-forward neural network language models, both trained on target n-grams only, and on "joint" or "bilingual" ngrams (Devlin et al., 2014;Le et al., 2012). For training these models we used the NPLM toolkit (Vaswani et al., 2013), for which we have now implemented gradient clipping to address numerical issues often encountered during training.

Baseline Features
We follow the standard approach to SMT of scoring translation hypotheses using a weighted linear combination of features. The core features of our model are a 5-gram LM score (i.e. log probability), phrase translation and lexical translation scores, word and phrase penalties, and a linear distortion score. The phrase translation probabilities are smoothed with Good-Turing smoothing (Foster et al., 2006). We used the hierarchical lexicalized reordering model (Galley and Manning, 2008) with 4 possible orientations (monotone, swap, discontinuous left and discontinuous right) in both left-to-right and right-to-left direction. We also used the operation sequence model (OSM) (Durrani et al., 2013) with 4 count based supportive features. We further employed domain indicator features (marking which training corpus each phrase pair was found in), binary phrase count indicator features, sparse phrase length features, and sparse source word deletion, target word insertion, and word translation features (limited to the top K words in each language, typically with K = 50).

Tuning
Since our feature set (generally around 500 to 1000 features) was too large for MERT, we used k-best batch MIRA for tuning (Cherry and Foster, 2012). To speed up tuning we applied threshold pruning to the phrase table, based on the direct translation model probability.

Decoding
In decoding we applied cube pruning (Huang and Chiang, 2007) with a stack size of 5000 (reduced to 1000 for tuning), Minimum Bayes Risk decoding (Kumar and Byrne, 2004), a maximum phrase length of 5, a distortion limit of 6, 100best translation options and the no-reorderingover-punctuation heuristic (Koehn and Haddow, 2009).

Finnish→English
Similar to last year , we built an unconstrained system for Finnish→English using data extracted from OPUS (Tiedemann, 2012). Our parallel training set was the same as we used previously, but the language model training set was extended with the addition of the news2015 monolingual corpus and the large WMT16 English CommonCrawl corpus. We used news-dev2015 for tuning, and newsdev2015 for testing during system development.
One clear problem that we noted with our submission from last year was the large number of OOVs, which were then copied directly into the English output. This is undoubtedly due to the agglutinative nature of Finnish, and probably was the cause of our system being poorly judged by human evaluators, despite having a high BLEU score. To address this, we split the Finnish input into subword units at both train and test time. In particular, we applied byte pair encoding (BPE) to split the Finnish source into smaller units, greatly reducing the vocabulary size. BPE is a technique which has been recently used to good effect in neural machine translation (Sennrich et al., 2016b), where the models cannot handle large vocbaularies. It is actually a merging algorithm, originally designed for compression, and works by starting with a maximally split version of the training corpus (i.e. split to characters) and iteratively merging common clusters. The merging continues for a specified number of iterations, and the merges are collected up to form the BPE model. At test time, the recorded merges are applied to the test corpus, with the result that there are no OOVs in the test data. For the experiments here, we used 100,000 BPE merges to create the model.
Applying BPE to Finnish→English was clearly effective at addressing the unknown word problem, and in many cases the resulting translations are quite understandable, e.g. source yös Intian on sanottu olevan kiinnostunut puolustusyhteistyösopimuksesta Japanin kanssa. base India is also said to be interested in puolustusyhteistyösopimuksesta with Japan. bpe India is also said to be interested in defence cooperation agreement with Japan. reference India is also reportedly hoping for a deal on defence collaboration between the two nations. However applying BPE to Finnish can also result in some rather odd translations when it overzealously splits: source Balotelli oli vielä kaukana huippuvireestään. base Balotelli was still far from huippuvireestään. bpe Baloo, Hotel was still far from the peak of its vitality. reference Balotelli is still far from his top tune.
We built four language models: an interpolated count-based 5-gram language model with all corpora, apart from the WMT16 CommonCrawl; separate count-based language models with WMT16 CommonCrawl and news2015; and a neural LM on news2015. A performance comparison across different language model combinations, and with and without BPE is shown in Table 1

Romanian→English
We trained our Romanian→English system using all data available for the constrained task. For system development, we split the newsdev2016 set into two parts randomly (so as to balance the "born English" and "born Romanian" portions), using one for tuning and one for testing. For building the final system, and for the contrastive experiments, we used the whole of newsdev2016 for tuning, and newstest2016 for testing.
In early experiments we noted that both the training and the development data were inconsistent in their use of diacritics leading to problems with OOVs and sparse statistics. To address this we stripped off all diacritics from the Romanian texts and the result was a significant increase in performance in our development setup. We also experimented with different language model combinations during development, with our submitted system using three different language model features: a neural LM trained on just news2015 monolingual, an n-gram language model trained on the WMT16 English CommonCrawl corpus, and a linear interpolation of language models trained on all other WMT16 English corpora.
In Table 1 we show how system performance varies under different language model combination and preprocessing conditions.

English→Romanian
For English→Romanian, we used all the data in the constrained track, including the Com-monCrawl language model data, and as with the Romanian→English system, we used news-dev2016 for the final tuning run.
The inconsistent use of diacritics in Romananian text also affected the English→Romanian system, however removing altogether would be problematic as we would then need a method for restoring them for the final system. So the only extra preprocessing we performed on the Romanian was to ensure that "t-comma" and "s-comma" were written correctly, with a comma rather than a cedilla.
Our final system used two different countbased 5-gram language models (one trained on all data, including the WMT16 Romanian Common-Crawl corpus, without pruning, and one trained on news2015 monolingual only), a neural language model trained on news2015 monolingual, and a bilingual language model trained on the parallel data, with source window of 15 and target window of 1. In Table 2 we show ablation experiments where we remove each of these language models.  Table 2: Effect of each of the language models used in the English→Romanian system. The experiments are not cumulative, so we first try pruning the "all" language model, then go back to the unpruned version and remove each LM in turn, observing the effect. The submitted system used all four LMs, and the scores shown are uncased BLEU scores on newstest2016.

English→German
For the English→German phrase-based system, we exploited several translation factors in addition to word surface forms, in particular: Och clusters (with 50 classes) and part-of-speech tags (Ratnaparkhi, 1996) on the English side, as well as Och clusters (50 classes), morphological tags, and part-of-speech tags on the German side (Schmid, 2000). Recent experiments for our IWSLT 2015 phrase-based system have reconfirmed that English→German translation quality can benefit from these factors when supplementary models over factored representations are used . For WMT16, we utilized the factors in the translation model, in operation sequence models, and in language models (for linearly interpolated 7-gram LMs over Och clusters and morphological tags).
Sparse source word deletion, target word insertion, and word translation features were integrated over the top 200 word surface forms and over selected factors (source and target Och clusters, source part-of-speech tags and target morphological tags). An unpruned 5-gram LM over words that was trained on all German data except the CommonCrawl monolingual corpus was supplemented by a separate pruned LM trained on the CommonCrawl data that had been provided as permissible data for the "constrained" track. Rather than applying a simple linear distortion score, we opted for sparse distortion features as described by Green et al. (2010), which we reimplemented in Moses. We activated sparse distortion features with a feature template based on jump distance, source part-of-speech tags, and target morpholog-ical tags.
The feature weights for our final system were tuned with hypergraph MIRA (i.e. batch MIRA over lattices representing the decoding search space) on a concatenation of newssyscomb2009 and newstest2008-2012.

German→English
For phrase-based translation from German, we applied syntactic pre-reordering (Collins et al., 2005) and compound splitting (Koehn and Knight, 2003) in a preprocessing step on the source side. The operation sequence model for the German→English phrase-based system was unpruned. We integrated three language models: an unpruned LM over all English data except the CommonCrawl monolingual corpus; a pruned LM over CommonCrawl; and a pruned LM over the monolingual News Crawl 2015 corpus. In addition to lexical smoothing with the standard lexicon models, we utilized a source-to-target IBM Model 1 (Brown et al., 1993) for sentence-level lexical scoring in a similar manner as described by Huck et al. (2011) for hierarchical systems. We tuned on the concatenation of newssyscomb2009 and newstest2008-2012.
Unlike last year's system -and different from the inverse translation direction (English→German)-we refrained from using any factors and instead set up a system that operates over surface form word representations only. In relation to last year's system, we were able to maintain high translation quality as measured in BLEU despite the abandonment of factors. However, we suspect that human judgment scores may suffer a bit from the abandonment of a factored model. We decided to drop the factored representations in favour of gains in decoding efficiency.
We furthermore did not employ any sparse features (sparse phrase length, source word deletion, target word insertion, or word translation features) in the German→English system since we did not observe any clear gains in preliminary experiments, and sparse features slow down tuning and decoding.
English→German and German→English translation results with our phrase-based systems are given in Table 3.

Spanish→English Biomedical
For our submission to the Spanish→English biomedical task, we created a parallel corpus using  all relevant data from WMT13, as well as the extra biomedical data provided by the task organisers, and the EMEA corpus from OPUS (Tiedemann, 2012). In total we had around 16M sentences of parallel data. Our monolingual corpus was made up of three parts: all the English monolingual medical data from WMT14 medical, WMT16 biomedical and EMEA (11M sentences); all the English LDC GigaWord data (180M sentences); and all the English general domain data from WMT16 (240M sentences). We used the monolingual data to build three different language models which were then linearly interpolated. System tuning was with the SCIELO development data provided for the biomedical task.

Syntax-based System Overview
For all syntax-based systems, we used a string-totree model based on a synchronous context-free grammar (SCFG) with linguistically-motivated labels on the target side.

Preprocessing
Except for English-Czech, which we describe separately in Section 5.1, preprocessing was similar to the phrase-based systems (Section 2.3). To parse the target-side of the training data, we used the Berkeley parser (Petrov et al., 2006;Petrov and Klein, 2007) for English, and the ParZu dependency parser (Sennrich et al., 2013) for German. Except where stated otherwise, we right-binarized the trees after parsing to increase rule coverage.

Word Alignment
As in the phrase-based models, we used fast_align for word alignment and the grow-diag-final-and heuristic for symmetrization.

Language Models
As in the phrase-based systems (Section 2.3), we used linearly-interpolated language models as standard, with some systems adding Common-Crawl and neural LMs. We detail the systemspecific combinations in Section 5.

Rule Extraction
SCFG rules were extracted from the word-aligned parallel data using the Moses implementation (Williams and Koehn, 2012) of the GHKM algorithm (Galley et al., 2004(Galley et al., , 2006. Minimal GHKM rules were composed into larger rules subject to restrictions on the size of the resulting tree fragment. We used the settings shown in Table 4  The parameters were relaxed for systems that used binarization to allow for the increase in tree node density. Further to the restrictions on rule composition, fully non-lexical unary rules were eliminated using the method described in Chung et al. (2011) and rules with scope greater than 3 (Hopkins and Langmead, 2010) were pruned from the translation grammar. Scope pruning makes parsing tractable without the need for grammar binarization.

Baseline Features
Our core set of string-to-tree feature functions is unchanged from previous years. It includes the ngram language model's log probability for the target string, the target word count, the rule count, and several pre-computed rule-specific scores. The rule-specific scores were: the direct and indirect translation probabilities; the direct and indirect lexical weights ; the monolingual PCFG probability of the tree fragment from which the rule was extracted; and a rule rareness penalty.

Decoding
Decoding for the string-to-tree models is based on Sennrich's (2014) recursive variant of the CYK+ parsing algorithm combined with LM integration via cube pruning (Chiang, 2007).

Tuning
The feature weights for the English→Czech and Finnish→English systems were tuned using the Moses implementation of MERT (Och, 2003). For the remaining systems we used k-best MIRA (Cherry and Foster, 2012) due to the use of sparse features.
We used randomly-chosen subsets of the previous years' test data to speed up decoding.

English→Czech
For English→Czech, we used Treex to preprocess and parse the Czech-side of the training data. Treex uses the MST parser (McDonald et al., 2005), which produces dependency graphs with non-projective arcs. In order to extract SCFG rules, we first applied the following conversion process: i) the dependency graphs were projectivized using the Malt Parser, which implements the method described in Nivre and Nilsson (2005) (we used the 'Head' encoding scheme); ii) the projective dependency graphs were converted to CFG trees. In addition, we reduced the complex positional tags to simple POS tags by discarding the morphological attributes. The CFG trees were not binarized.
We also experimented with unificationbased agreement and case government constraints (Williams and Koehn, 2011;Williams, 2014). Specifically, our constraints were designed to enforce: i) case, gender, and number agreement between nouns and pre-nominal adjectival modifiers; ii) number and person agreement between subjects and verbs; iii) case agreement between prepositions and nouns; iv) use of nominative case for subject nouns. For every Czech word in the training data, we obtained a set of morphological analyses using MorphoDiTa (Straková et al., 2014). From these analyses, we constructed a lexicon of feature structures. For constraint extraction, we used handwritten rules along the lines of those described in Williams (2014).
In preliminary experiments we used a smaller training set, comprising 2 million sentence pairs sampled from OPUS and monolingual data from last year's WMT translation task. We used two test sets from the HimL project and the Khresmoi test set. Results with and without constraints are shown in Table 5. We used hard constraints and reused the baseline weights (re-tuning did not appear to give additional gains  Although the gains in BLEU were small, previous analysis for German showed that BLEU lacks sensitivity to grammatical improvements when compared to human evaluators (Williams, 2014).
We trained the final system on all of the provided training and monolingual data. In addition to the interpolated LM, we used a model trained on the CommonCrawl data. Results are shown in Table 6.   system  BLEU  2015 2016  baseline 17.3 20.1 + constraints 17.5 20.2 + CC LM 17.9 20.9 Table 6: Translation results on the final system for English→Czech with unification-based constraints. Cased BLEU scores are shown. Note that baseline weights are reused in the experiments with constraints.

Manual Analysis
We carried out a small manual analysis of the submitted system with and without unification-based constraints (the CC LM was used in both cases). In order to remove the effect of tuning variance, we used the same model weights in both cases (the weights were learned on the version without constraints). The BLEU scores of the two systems were 20.9 (with constraints) and 20.7 (without constraints). A large majority of the outputs (81% of the 2999 sentences in the newstest2016) are identical. Looking at a sample of 100 sentences with some differences, we classified differring areas to see in what aspects the outputs of the two systems differ. In total, there were 104 such areas (some sentences had more than one area of interest). Table 7 summarizes the overall evaluation of these areas (the annotation was not blind, we knew which system was which). The majority of the areas were of an equal quality, in fact equally bad overall, so neither of the compared systems delivered an acceptable translation.

Much
Crazy Better Better Equal Worse Reordering 4 41 44 12 3 Table 7: Manual evaluation of translations as proposed by the English→Czech system with unification constraints vs. the same system without constraints.
In 4 cases, the system with constraints delivered much better translation, and three of those were overall improvement of the sentence structure.
In 41 cases, the area was better for various reasons. Most frequently (16 cases), this was indeed the agreement within noun and prepositional phrases (adjective matching in case the preposition etc.). In 9 additional cases, the NP or PP was better translated but in other aspects than morphological case, number of gender. For instance the baseline system translated the phrase "between the departments of individual hospitals" as "between the individual departments of the hospitals" (in morphologically well-formed Czech). Beyond better NPs and PPs, the constraints have also helped overall sentence or clause structure (5 cases), lexical choice (4 cases) and verbs and their belongings (2 cases).
In 15 cases, the constraints forced the system to select a worse translation, damaging sentence structure, lexical choice, spuriously introducing negation etc. We highlight 3 of these cases, where the system with constraints accidentally moved words far away from their correct location ("Crazy Reordering" in Table 7). This suggests that due to sparse data, the application of constraints should  be better balanced with respect to other parts of the model. In constrast to German, targetting Czech usually does not need long-distance reordering and doing it risks more serious translation errors than sticking to the English word order.
Since the hard unification constraints effectively only avoid some of the possible translations (i.e. reduce the search space), we conclude that having to obey mere agreement constraints helps to select a hypothesis better in a surprisingly larger span of words, improving overall sentence structure on average.

English→German
This year's string-to-tree submission for English→German is similar to last year's system . In addition to the baseline feature functions, it contains count-based 5-gram Neural Network language model (NPLM) (Vaswani et al., 2013), a relational dependency language model (RDLM) (Sennrich, 2015), and soft source-syntactic constraints . The parameters of the model are tuned towards the linear interpolation of BLEU and the syntactic metric HWCM (Liu and Gildea, 2005;Sennrich, 2015). Trees are transformed through binarization and a hierarchical representation of morphologically complex words (Sennrich and Haddow, 2015).
For the soft source-syntactic constraints, we annotate the source text with the Stanford Neural Network dependency parser (Chen and Manning, 2014), along with heuristic projectivization (Nivre and Nilsson, 2005).
Results are shown in Table 8. We report results of last year's system , which was ranked (joint) first at WMT 15. Our improvements this year stem from particle verb restructuring (Sennrich and Haddow, 2015), and the use of the new monolingual News Crawl 2015 corpus for the Kneser-Ney language model. 2

Finnish→English
Our Finnish→English syntax-based system was similar to last year's . The main difference from the basic setup of Section 4 is that we preprocessed the Finnish data to segment words into morphemes. We also added a CommonCrawl language model in addition to the interpolated LM.
For segmentation, we used Morfessor 2.0 with default settings, first training a segmentation model, then using it to segment all words in the source-side training and test data. Morfessor takes a set of word types as input and we found that it was important for translation quality to use a large training vocabulary. Table 9 gives mean BLEU scores for this setup, averaged over three MERT runs. Our baseline is the standard stringto-tree setup (i.e. without segmentation and without the CommonCrawl LM). For segmentation, we experimented with varying amounts of training data, initially using the Finnish side of the provided parallel corpora, then adding the monolingual Finnish data (apart from CommonCrawl), and finally adding 10% of the CommonCrawl vocabulary (we extracted the full vocabulary from Com-monCrawl and then randomly sampled 10%). We found that using larger amounts of training data was prohibitively slow.

German→English
For German→English we built a string-to-tree system with a similar setup to last year's . In addition we used sparse features to determine the non-terminal labels for un-system BLEU dev test baseline (phrase-structure) 28.6 33.5 + NER before split 28.8 33.8 + CommonCrawl LM* 29.4 34.4 contrastive (dependency) + NER before split 28.1 33.0 Table 10: Translation results of German→English string-to-tree translation system on dev (news-test2015) and test (newstest2016). *submitted system.
known words similar to the English→German systems described by  and . We also tagged named entities to avoid over-splitting of compounds. For example the script provided with Moses for compound splitting will split Florstadt nach Bad Salzhausen into flor Stadt nach Bad Salz hausen. This is then wrongly translated by the baseline system as Flor after bath salt station. We applied a 3-class named entity tagger (Finkel et al., 2005;Faruqui and Padó, 2010) on the German side of the corpus prior to splitting and removed the annotations afterwards. We also trained a contrastive system with target-side dependency relations instead of PTB-style phrase-structures. The English side of the parallel corpora was annotated with the Stanford Neural Network dependency parser (Chen and Manning, 2014), along with heuristic projectivization (Nivre and Nilsson, 2005) and headbinarization (Sennrich and Haddow, 2015). We report the cased BLEU scores for different setups of our system in Table 10.

Romanian→English
For Romanian→English we built a string-to-tree system similar to the German→English system. However we did not use compound splitting and we allowed glue rules. Similar to the phrase-based setup we used half of the newsdev2016 for tuning and the other half as development set. We normalized the corpora by removing all diacritics from the Romanian side. We report the cased BLEU scores for different setups of our system in Table 11.