{'fraction_non_alphanumeric': 0.04478792235801582, 'fraction_numerical': 0.015959741193386053, 'mean_word_length': 4.612467218075449, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Latent Domain Word Alignment for Heterogeneous Corpora

This work focuses on the insensitivity of existing word alignment models to domain differences, which often yields suboptimal results on large heterogeneous data. A novel latent domain word alignment model is proposed, which induces domain-conditioned lexical and alignment statistics. We propose to train the model on a heterogeneous corpus under partial supervision, using a small number of seed samples from different domains. The seed samples allow estimating sharper, domain-conditioned word alignment statistics for sentence pairs. Our experiments show that the derived domain-conditioned statistics, once combined together, produce notable improvements both in word alignment accuracy and in translation accuracy of their resulting SMT systems.


Introduction
Word alignment currently constitutes the basis for phrase extraction and reordering in phrase-based systems, and its statistics provide lexical parameters used for smoothing the phrase pair estimates. For over two decades since IBM models (Brown et al., 1993) and the HMM alignment model (Vogel et al., 1996), word alignment remains an active research line, e.g., see recent work (Simion et al., 2013;Tamura et al., 2014;Chang et al., 2014).
During the past years we witnessed an increasing need to collect and use large heterogeneous parallel corpora from different domains and sources, e.g., News, Wikipedia, Parliament Proceedings. It is tacitly assumed that assembling a larger corpus should improve a phrase-based system coverage and performance. Recent work (Sennrich et al., 2013;Carpuat et al., 2014;Cuong and Sima'an, 2014b;Kirchhoff and Bilmes, 2014;Cuong and Sima'an, 2014a) shows that this is not necessarily true as phrase translations as well as (bi-and monolingual) word co-occurrence statistics could differ across domains. This suggests that the word alignment quality obtained from IBM and HMM alignment models might also be affected in heterogeneous corpora.
Intuitively, in heterogeneous data certain words are present across many domains, whereas others are more specific to few domains. This suggests that the translation probabilities for words will be as fractioned as the diversity of its translations across the domains. Furthermore, because the IBM and HMM alignment models use context-insensitive conditional probabilities, in heterogeneous corpora the estimates of these probabilities will be aggregated over different domains. Both issues could lead to suboptimal word alignment quality.
Surprisingly, the insensitivity of the existing IBM and HMM alignment models to domain differences has not received much attention thus far (see the study of Bach et al. (2008) and Gao et al. (2011) for reference in the literature). We conjecture that this is because it is not fully clear how to define what constitutes a (sub)-domain. In this paper we propose to exploit the contrast between the alignment statistics in a handful of seed samples from different domains in order to induce domain-conditioned probabilities for each sentence pair in the heterogeneous corpus. Crucially, some sentence pairs will be more similar to a seed domain than others, whereas some sentence pairs might be dissimilar to all seed domains. The number and choice of seed domains depends largely on the available resources but intuitively these seed domains are chosen to be relevant to parts of the heterogeneous corpus. A small number of such seeds can be expected to notably improve word alignment accuracy. In fact, a single seed sample already allows us to exploit the contrast between two parts in the corpus: similar or dissimilar to the seed data.
Considering the small seed samples as partial supervision, in this paper we explore the question: how to obtain better word alignment in a heterogeneous, mix-of-domains corpus? We present a novel latent domain HMM alignment model, which aims to tighten the probability estimates of the generative alignment process of a sentence pair, and of the probability estimates of the sentence pair itself for a specific domain. We also present an accompanying training regime guided by partial supervision using the seed samples, exploiting the contrast between the domain-conditioned alignment statistics in these samples. This way we aim for an alignment model that is more domain-sensitive than the original HMM alignment model. Once the domainconditioned statistics are induced, we discuss how to combine them together to express the probability of a sentence pair as a mixture over specific domains.
Finally, we report experimental results over heterogeneous corpora of 1M, 2M and 4M sentence pairs, where we are provided domain information for different samples of 10%, 5% and 2.5% of the heterogeneous data respectively. A large number of experiments are reported, showing that the latent domain HMM model produces notable improvements in word alignment accuracy over the original HMM alignment model. Furthermore, the translation accuracy of the resulting SMT systems is significantly improved across four different translation tasks.

HMM Alignment Model
In this section, we briefly review the HMM alignment model (Vogel et al., 1996). The generative story of the model is shown in Figure 1. The latent states take values from the target language words and generate source language words. Formally, we use e = (e 1 , . . . , e I ) to denote the target sentence with length I and f = (f 1 , . . . , f J ) Observed layer (source words) Latent alignment layer (target words) Figure 1: HMM alignment model with observed and latent alignment layers.
to denote the source sentence with length J. For an alignment a = (a 1 , . . . , a J ) of a sentence pair e, f , the model factors P (f, a| e) into the word translation and transition probabilities: Here, P (f j | e a j ) represents the word translation probabilities and P (a j | a j−1 ) 1 represents the transition probabilities between positions. Note that P (a j | a j−1 ) depends only on the distance (a j − a j−1 ). Note also that the first-order dependency model is an extension of the uniform dependency model and zero-order dependency model of IBM models 1 and 2, respectively. In this work, we model explicitly distances in the range ±5. Note that null-links are also explicitly added in our implementation, following Och and Ney (2003) and Graca et al. (2010).
Once the HMM alignment model is trained, the most probable alignment,â for each sentence pair can be computed by:â = argmax a P (f, a| e). Here, the search problem can be solved by the Viterbi algorithm.

Latent Domain HMM Alignment Model
Because the heterogeneous data contains a mix of diverse domains, the induced statistics derived from word alignment models reflect translation preferences aggregated over these domains. In this sense, they can be considered domain-confused statistics (Cuong and Sima'an, 2014a). This work thus focuses on more representative statistics: the domainconditioned word alignment statistics, i.e., the statistics with respect to each of the diverse domains.
By introducing a latent variable D representing domains of the heterogeneous data, we aim  (2) The generative story of the model is shown in Figure 2. Note how domain-conditioned alignment statistics, P (·| ·, D) contain their former domainconfused alignment statistics, P (·| ·) as special case f P (f j |e a j )P (D|f j , e a j ) , P (a j |a j−1 , D) = P (a j |a j−1 )P (D|a j , a j−1 ) a j P (a j |a j−1 )P (D|a j , a j−1 ) .
With an additional latent domain layer, it becomes crucial to train the model in an efficient way. As suggested by Eq. 3 and 4, we could simplify training by breaking up the estimation process into two steps. That is, we train alignment parameters, P (·| ·) or domain parameters, P (D| ·, ·) first, hold them fixed before training the other kind of the parameters. 3 Instead, in this work we design an algorithm that trains both of them simultaneously via training domain-conditioned parameters P (·| , ·, D) directly.

Training
Basically, our model can be viewed as having a set, Θ of N subsets of domain-conditioned parameters, Θ D for N different domains, i.e., Θ = {Θ D 1 , . . . , Θ D N }. In this work, to simplify the learning problem we assume that the domains are very different from each other. If this assumption does not hold, the learning problem would shift from single-label learning to multiple-label learning. We leave this extension for future work.
Our training procedure seeks the parameters Θ that maximize the log-likelihood, L of the data: There, however, does not exist a closed-form solution for maximizing L, and EM comes as an alternative solution to fit the model. EM maximizes L via blockcoordinate ascent on a "free energy" lower bound F(q, Θ) (Neal and Hinton, 1999), using an auxiliary distribution q over both the latent variables: In the E-step of the EM algorithm, we fix Θ and aim to find the distribution q * that maximizes F(q, Θ) over the heterogeneous data. Simple math- is the Kullback-Leiber divergence between two distributions. The distribution q * can be thus derived as Here, P Θ D (D| f, e) aims to exploit the contrast between the domain-sensitive alignment statistics.
Assigning higher probability to one domain forces lower probability assignment to other domains. Note that P Θ D (f, a| e, D) is given in Eq. 2 and a P Θ D (f, a| e, D) can be computed efficiently using dynamic programming. 4 Meanwhile, P Θ D (D| f, e) can be derived by Bayes' rule, i.e., Here, the estimation of the domain prior parameters is easy, f,e c(D; f, e) Figure 3: Pseudocode for the training algorithm for the latent domain HMM alignment model. Note that notation P (c) denotes current iteration estimates, and P (+) denotes the re-estimates.
generative process for every sentence pair in the heterogeneous data with respect to a specific domain. Following (Cuong and Sima'an, 2014b), we factor it into two kinds of models in a symmetrized strategy: can be thought of as the domain-conditioned translation models, aiming to model how well a target/source sentence is generated over a source/target sentence with respect to a domain. 5 Meanwhile, P Θ D (·| D) can be thought of as the domain-conditioned language models (LMs), aiming to model how fluent a source/target sentence with respect to a domain. For simplicity, once the domain-conditioned LMs are trained, they will stay fixed during training, i.e., LM probabilities are not parameters in our model.
In the M-step of the EM algorithm, we fix the derived q * and aim to find the parameter set Θ * that maximizes F(q, Θ) over the data. This can be (easily) done by using q * to softly fill in the values of a and D to estimate model parameters.

Pseudocode
In summary, the model has three kinds of parameters -word translation, word transition, and domain prior parameters. We now summarize the training via presenting the pseudocode.
First, we present expected count notations with respect to domains for the parameters. We use c(f | e; f, e, D) to denote the expected counts that word e aligns to word f . We use c(i| i ; f, e, D) to denote the expected counts that two certain con-5 Note that PΘ D (·| ·, D) = a PΘ D (·, a| ·, D) and it can be thus computed efficiently using dynamic programming. secutive source words j and j − 1 align to two target words i and i respectively, i.e., j aligns to i and j −1 aligns to i . Finally, we also use c(D; f, e) to denote the expected count of domain priors. Note that all the expected counts are in the translation (f| e). Figure 3 represents the pseudocode.

Learning with Partial Supervision
We now discuss remaining issues on how to guide the learning with partial supervision, i.e., how to use the given domain information of seed samples to guide the learning.

Number of Domains
The values of D ∈ [1..(N + 1)] depends on the N available seed samples plus the so-called "out-domain," i.e., the part of the heterogeneous data that is dissimilar to all of the N sample domains.
Parameter Initialization We first discuss how to initialize the domain prior parameters. If a sentence pair f, e belongs to a sample with a pre-specified domain D i , we initialize P (D i | f, e) close to 1, and, Furthermore, we uniformly create the domain prior parameters for the rest of sentence pairs. Uniform initialization for the domain-conditioned alignment parameters is also a reasonable option. Nevertheless, a more effective way is to make use of the domain-specific seed samples and the pool of the rest sentence pairs in the heterogeneous data. 6 That is, we train the model on each of the samples, assign-ing the derived probabilities as the initialization for their corresponding domain-conditioned alignment parameters. In our implementation, one EM iteration is usually dedicated for this. It should be noted that we ignore the domain prior parameters in the model during the period.
Parameter Constraints During training, it would be also necessary to keep the domain prior parameters fixed for all sentence pairs that belong to seed samples. This can be thought of as the constraints derived from the partial knowledge, guiding the learning to a desirable parameter space.
Domain-conditioned LMs training We now discuss how to train the domain-conditioned LMs with partial supervision. It would be reasonable to use the domain-specific seed samples to train their exemplifying domain-conditioned LMs, and the pool of the rest sentence pairs to train the out-domain LMs. Nevertheless, the out-domain LMs trained on such a big corpus could dominate the other domain-conditioned LMs. Following Cuong and Sima'an (2014b), we rather create a "pseudo" outdomain sample to train the out-domain LMs, i.e., the creation is via an inspired burn-in period. In brief, an EM iteration is dedicated just to compute P (D OU T | f, e) for all sentences, ranking them and select a small subset with highest score as the (on the fly) pseudo out-domain sample.
Note that our partial learning framework is very simple. There are various advanced learning framework that are also applicable with the partial supervision, e.g., Posterior Regularization . This leaves much space for future work.

Domain-conditioned Decoding
At test time, assigning each sentence pair to a single most likely domain (hard decision) is likely to result in sub-optimal performance. 7 Instead we average over domains (soft decision) while predicting the translation. Formally for each sentence pair, e, f , we can find their best Viterbi alignment,â as 7 Later experiments on word alignment will confirm this. Here, we derive the last equation by applying Bayes' rule to P (D| e), i.e., P (D| e) ∝ P (e| D)P (D). Interestingly, our Viterbi decoding now relies on a mix of domain-conditioned statistics for each sentence pair. The computing of term D (a) for all possible alignments, a, however, is intractable, making the search problem difficult. Inspired by Liang et al. (2006), we opt instead for a heuristic objective function as follows 8 : . (5) Here, note that p is a lower bound for p, when 0 ≤ p ≤ 1, according to Jensen's inequality. With Eq. 5, it is straightforward to design a dynamic programming algorithm to decoding, e.g., the Viterbi algorithm. In practice, we observe that the approximation yields good results. Later experiments on word alignment will present this in detail.

Experimental Setup
In the following experiments, we use three heterogeneous English-Spanish corpora consisting of 1M , 2M and 4M sentence pairs respectively. These corpora combine two parts. The first part respectively 0.7M , 1.7M and 3.7M is collected from multiple domains and resources including EuroParl (Koehn, 2005), Common Crawl, United Nation, News Commentary. The second part consists of three domainexemplifying samples consisting of roughly 100K sentence pairs for each one (total 300K). Each of these three samples (manually collected by a commercial partner) exemplifies a specific domain related to Legal, Hardware and Pharmacy.
Outlook In Section 7 we examine the word alignment yielded by the HMM alignment model and our latent domain HMM alignment model. In Section 8 we proceed further to examine the translation produced by derived SMT systems.

Word Alignment Experiment
For alignment accuracy evaluation, we use a data set of 100 sentence pairs with their "golden" alignment from Graca et al. (2008). Here, the golden alignment consists of sure links (S) and possible links (P ) for each sentence pair. Counting the set of generating alignment links (A), we report the word alignment accuracy by precision ( |A∩P | |P | ), recall ( |A∩S| |S| ), alignment error rate (AER) (1 − |A∩P |+|A∩S| |A|+|S| ) (Och and Ney, 2003). 9 For all experiments, we use the same training configuration for both the baseline/the latent domain alignment model: 5 iterations for IBM model 1/the latent domain model; 3 iterations for HMM alignment model/the latent domain model. For evaluation, we first align the sentence pairs in both directions and then symmetrize them using the growdiag-final heuristic (Koehn et al., 2003).
For reference we also report the performance of a considerably more expressive Model 4, capable of capturing more structure, but at the expense of intractable inference. Using MGIZA++ (Gao and Vo-gel, 2008), we run 5 iterations for training Model 1, 3 iterations for training the HMM alignment model, Model 3 and Model 4.

Learning with Single Domain
We first examine the binary case, where we are given domain information in advance for each kind of samples only, e.g., Legal, or Pharmacy, or Hardware. For the different sizes of the heterogeneous data (1M , 2M and 4M ) the seed sample size is thus 10%, 5% and 2.5% respectively. Note that in such cases, training the latent domain alignment model induces two domain-conditioned statistics: in-domain vs. out-domain (D 1 and D 2 respectively). Once the model is trained, we combine the induced domain-conditioned statistics together (Eq. 5) and examine the produced word alignment output. Table 1 presents the results. Most importantly, it shows that as long as providing domain information for reasonably large enough data, learning the latent domain alignment model notably improves the word alignment accuracy. For instance, given in advance the domain information for a sample of 10%, and 5% of the heterogeneous corpora, our model consistently improves the word alignment accuracy in all cases. Meanwhile, given in advance the domain information for a relatively small sample of 2.5% of the heterogeneous data, the results are mixed. We obtain a good performance/slightly better performance/worse performance with the case of Hardware/Legal/Pharmacy respectively.

What do domain-conditioned statistics look like?
To have an idea what the induced statistics look like, we investigate their conditional entropy. Here, we present the conditional entropy for the domain-confused/-conditioned word translation statistics induced from the HMM alignment model/its latent domain model. Note that similar results are observed for transition tables.  Formally, for a translation table, F, E , its conditional entropy, H(F | E) can be estimated from its possible word pairs, e, f : H(F | E) = − e P (e) f P (f | e) log P (f | e). Table 2 reveals that the induced D 1 -conditioned statistics need much less bits to represent than the induced domain-confused statistics, e.g., 1124.43, 1104.58, 1115.52 vs. 1348.53. This implies the induced D 1conditioned statistics are much more predictable compared to the domain-confused statistics. Meanwhile, the induced D 2 -conditioned statistics are similar to the domain-confused statistics in terms of the conditional entropy, e.g., 1354.58, 1385.35, 1342.54 vs. 1348.53.

Learning with Multiple Domains
It would be more interesting to learn the latent domain alignment model for multiple domains, rather than learning with each of them separately. In detail, using all the seed samples from different domains, we aim to learn four different domain-conditioned statistics simultaneously. Under this setting, we obtain good results, as described in Table 1. For the two cases with the training corpora of 2M and 4M sentence pairs respectively, learning with the combining domain prior knowledge produces the best word alignment accuracy compared to the rest. In the last case with the training corpus of 1M sentence pairs, learning with the combining domain prior knowledge produces compatible with the case of Hardware, i.e., the best binary domain case.
Table 1 also reveals that the performance of our model approaches Model 4, even though Model 4 is much more complex and computationally expensive.

Domain-conditioned statistics combination
We also investigate the relation between the number of domain-conditioned statistics "involved" in the Viterbi decoding (Eq. 5) and the word alignment accuracy. Table 3 presents the results in case of using only the induced D 1 -/, D 2 -/, D 3 -/, D 4conditioned statistics separately, and also using their different combinations. Interestingly, we observe that using more domain-conditioned statistics for decoding incrementally improves the word alignment accuracy over the heterogeneous data. While the domain-conditioned statistics are very different in their characteristics from each other, the results reveal how they are complementary to the others, conveying a mix of domains for each sentence pair.  Finally, it is also tempting to make a comparison between the hard vs. soft domain assignment in Viterbi decoding. Here, for hard domain decision we simply do decoding with the following objec-tive function:â = argmax a P (f, a| e,D), wherê D = argmax D P (D| e). Table 3 presents the results. It reveals that a soft domain assignment on the domain of sentence pairs results in a better alignment accuracy than a hard domain assignment. 10

Translation Experiment
In this section, we investigate the contribution of our model in terms of the translation accuracy. Here, we run experiments on the heterogeneous corpora of 1M, 2M, and 4M sentence pairs, testing the translation accuracy over four different domain-specific test sets related to News, Pharmacy, Legal, and Hardware.
We use a standard state-of-the-art phrase-based system as the baseline. Our dense features include MOSES (Koehn et al., 2007) baseline features, plus hierarchical lexicalized reordering model features (Galley and Manning, 2008), and the word-level feature derived from IBM model 1 score, c.f., (Och et al., 2004). 11 The interpolated 5-grams LMs with Kneser-Ney are trained on a very large monolingual corpus of 2B words. We tune the systems using kbest batch MIRA (Cherry and Foster, 2012). Finally, we use MOSES (Koehn et al., 2007) as decoder.
Our system has exactly the same setting with the baseline, except: (1) To learn the translation, we use the alignment result derived from our latent domain HMM alignment model, rather than the HMM alignment model; and (2) We replace the word-level feature with our four domain-conditioned word-level features derived from the latent domain IBM model 1. Here, note that our latent model is learned with the supervision from the combining domain knowledge of all three domain-specific seed samples. 10 Note that similar results are also observed for training, in which a soft domain assignment using soft EM produces better alignment accuracy than a hard domain assignment using hard EM. (See (Gao et al., 2011) for reference to hard domain assignment to training data.) This is perhaps due to the characteristics of the data we use. For instance, News sentence pairs are useful for translating Legal,Financial or EuroParl to varying degrees. 11 For every phrase pair f ,ẽ with their length of mf and lẽ respectively, the lexical feature estimates a probability in Model 1 style between their word pairs fj, ei (i.e. P (f |ẽ) = lẽ mf j=1 lẽ i=1 P (fj|ei)). Note that adding word-level features from both translation sides does not help much, as observed by (Och et al., 2004). We thus add only an one from a translation side.   (Bojar et al., 2013). For the Pharmacy, Legal, and Hardware translation tasks, we tune systems on three domain-specific dev sets of 1, 000 sentence pairs and test them on three domain-specific test sets of 1, 016, 1, 326 and 1, 721 sentence pairs. We report three metrics -BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2011) and TER (Snover et al., 2006), with statistical significance at 95% confidence interval under paired bootstrap re-sampling. 12 For every system reported, we run the optimizer three times, before running MultEval (Clark et al., 2011) for resampling and significance testing.
Results are in Table 4, showing significant improvements across four different test sets over different heterogeneous corpora sizes. Table 5 gives a summary of the improvements. On average, over heterogeneous corpora of 1M, 2M and 4M sentence pairs, our system outperforms the baseline by 1.0 BLEU, 1.4 BLEU and 0.7 BLEU, respectively. Finally, we observe that our system produces comparably good performance to the MGIZA++-based system. When 1M data is considered, on three of four tasks, our system produces at least compatible translation accuracy to the corresponding MGIZA++-based system. Further analysis reveals that the improvement is due to not only the reduction in alignment error rate, but also the use of the domain-sensitive lexical features. Moreover, the domain-sensitive lexical features is particularly useful when the domain of the test data matches with the domain of seed samplers. This is also widely observed in the literature, e.g., see (Eidelman et al., 2012;Hasler et al., 2014;Hu et al., 2014).

Related Work and Conclusion
In terms of domain-conditioned statistics for word alignment, a distantly related research line (Tam et al., 2007;Zhao and Xing, 2008) focuses on using document topics to improve the word alignment. In terms of learning word alignment with partial supervision, another distantly related research line focuses on semi-supervised training with partial manual alignments (Fraser and Marcu, 2006;. Finally, recent work also focuses on data selection (Kirchhoff and Bilmes, 2014;Cuong and Sima'an, 2014b), mixture models (Carpuat et al., 2014), instance weighting (Foster et al., 2010) and latent variable models (Cuong and Sima'an, 2014a) over heterogeneous corpora.
One main contribution of this work is the novelty of exploring the quality of word alignment in heterogeneous corpora. This, surprisingly, has not received much attention thus far (see the study of Bach et al. (2008) and Gao et al. (2011) for reference in the literature). Another major contribution of this work is a learning framework for latent domain word alignment with partial supervision using seed domains. We present its benefits for improving not only the word alignment accuracy, but also the translation accuracy resulting SMT systems produce. We hope this study sparks a new research direction for using domain samples, which is cheap to gather, but has not been exploited before.
One obvious direction for future work might be to integrate the model into fertility-based alignment models (Brown et al., 1993), as well as other recently advanced alignment frameworks, e.g., (Simion et al., 2013;Tamura et al., 2014;Chang et al., 2014). Another interesting direction might be to integrate our model into advanced mixing multiple translation models, improving SMT systems trained on the heterogeneous data (Razmara et al., 2012;Sennrich et al., 2013;Carpuat et al., 2014). Finally, an open question is whether it is possible to learn the latent domain alignment model in a fully unsupervised style. This challenge deserves more attention in future work.