{'fraction_non_alphanumeric': 0.03745839636913767, 'fraction_numerical': 0.018154311649016642, 'mean_word_length': 4.796562609610663, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 2, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation

Despite some empirical success at correcting exposure bias in machine translation, scheduled sampling algorithms suffer from a major drawback: they incorrectly assume that words in the reference translations and in sampled sequences are aligned at each time step. Our new differentiable sampling algorithm addresses this issue by optimizing the probability that the reference can be aligned with the sampled output, based on a soft alignment predicted by the model itself. As a result, the output distribution at each time step is evaluated with respect to the whole predicted sequence. Experiments on IWSLT translation tasks show that our approach improves BLEU compared to maximum likelihood and scheduled sampling baselines. In addition, our approach is simpler to train with no need for sampling schedule and yields models that achieve larger improvements with smaller beam sizes.


Introduction
Neural machine translation (NMT) models are typically trained to maximize the likelihood of reference translations (Sutskever et al., 2014;Bahdanau et al., 2015). While simple and effective, this objective suffers from the exposure bias problem (Ranzato et al., 2015): the model is only exposed to reference target sequences during training, but has to rely on its own predictions at inference. As a result, errors can accumulate along the generated sequence at inference time. This is a well-known issue in sequential decision making (Langford and Zadrozny, 2005;Cohen and Carvalho, 2005;Kääriäinen and Langford, 2006, i.a.) and it has been addressed in past work by incorporating the previous decoding choices into the training scheme, using imitation learning (Daumé et al., 2009;Ross et al., 1 The code is available at https://github.com/ Izecson/saml-nmt 2011; Leblond et al., 2018) and reinforcement learning (Ranzato et al., 2015;Bahdanau et al., 2016) techniques. In this paper, we focus on a simple and computationally inexpensive family of approaches, known as Data as Demonstrator (Venkatraman et al., 2015) and scheduled sampling Goyal et al., 2017). The algorithms use a stochastic mixture of the reference words and model predictions with an annealing schedule controlling the mixture probability. Despite their empirical success in various sequence prediction tasks, they are based on an assumption that does not hold for machine translation: they assume that words in the reference translations and in sampled sequences are aligned at each time step, which results in weak and sometimes misleading training signals.
In this paper, we introduce a differentiable sampling algorithm that exposes machine translation models to their own predictions during training, and allows for differences in word order when comparing model outputs with reference translations. We compute the probability that the reference can be aligned with the sampled output using a soft alignment predicted based on the model states, so that the model will not be punished too severely for producing hypotheses that deviate from the reference, as long as the hypotheses can still be aligned with the reference.
Experiments on three IWSLT tasks (German-English, English-German and Vietnamese-English) show that our approach significantly improves BLEU compared to both maximum likelihood and scheduled sampling baselines. We also provide evidence that our approach addresses exposure bias by decoding with varying beam sizes, and show that our approach is simpler to train than scheduled sampling as it requires no annealing schedule.

Approach
Our approach is designed to optimize the standard sequence-to-sequence model for translating a source sentence x into a target sentence y (Bahdanau et al., 2015). This model computes the probability of y given x as: where θ represents the model parameters. Given x, the model first produces a sequence of hidden representations h 1...T : where T is the length of y, and f is usually an encoder-decoder network. At each time step t, the hidden representation h t is fed to a linear projection layer s t = W h t + b to obtain a vector of scores s t over all possible words in the vocabulary V. Scores are then turned into a conditional probability distribution: p(· | y <t , x; θ) = softmax(s t ).
The traditional maximum likelihood (ML) objective maximizes the log-likelihood of the training data D ≡ {(x (n) , y (n) )} N n=1 consisting of N pairs of source and target sentences: At test time, prefixes y <t are subsequences generated by the model and therefore contain errors. By contrast, in ML training, prefixes y <t are subsequences of reference translations. As a result, the model is never exposed to its own errors during training and errors accumulate at test time. This mismatch is known as the exposure bias problem (Ranzato et al., 2015).  introduced the scheduled sampling algorithm to address exposure bias. Scheduled sampling gradually replaces the reference words with sampled model predictions in the prefix used at training time. An annealing schedule controls the probability of using reference words vs. model predictions. The training objective remains the same as the ML objective, except for the nature of the prefixŷ <t , which contains a mixture of reference and predicted words:

Limitations in Scheduled Sampling
Despite the empirical success of scheduled sampling, one limitation is that the discontinuity of the argmax operation makes it impossible to penalize errors made in previous steps, which can lead to slow and unstable training. We address this issue using a continuous relaxation to the greedy search and sampling process, similarly to Goyal et al. (2017), which we describe in Section 2.2.
Another limitation of scheduled sampling is that it incorrectly assumes that the reference and predicted sequence are aligned by time indices which introduces additional noise to the training signal. 2 We address this problem with a novel differentiable sampling algorithm with an alignment based objective called soft aligned maximum likelihood (SAML). It is used in combination with maximum likelihood to define our training objective J = J M L + J SAM L , where J M L is computed based on reference translations, and J SAM L is computed based on sampled translations of the same input sentences. We define J SAM L in Section 2.3.

Differentiable Sampling
To backpropagate errors made in the previous decoding steps, we use a continuous relaxation of the discrete sampling operation similar to Goyal et al. (2017), except that we use the Straight-Through (ST) Gumbel-Softmax estimator (Jang et al., 2017;Bengio et al., 2013) instead of Gumbel-Softmax (Jang et al., 2017;Maddison et al., 2014) to better simulate the scenario at inference time. 3 The Gumbel-Softmax is derived from the Gumbel-Max trick (Maddison et al., 2014), an algorithm for sampling one-hot vector z ∈ R k from a categorical distribution (p 1 , ..., p k ): where g i is the Gumbel noise drawn i.i.d from Gumbel(0, 1) 4 , and β is a hyperparameter controlling the scale of the noise. Here, the trick is used to approximate the discontinuous argmax function with the differentiable softmax: We make dinner </s> We make </s> will log p("dinner" | "We will", x) log [ a13 p("dinner" | x) + … + a43 p("dinner" | "We will make", x) + a53 p("dinner" | "We will make dinner", x) ] (b) SAML Objective Figure 1: Difference between objectives used in scheduled sampling (left) and our approach (right), when computing the contribution to the objective of the reference word "dinner". The schedule sampling hypothesis uses a mixture of the reference (black) and sampled (blue underlined) words, while the entire hypothesis sequence is sampled in our approach.
where τ is the temperature parameter. As τ diminishes to zero,z becomes the same as one-hot sample z.
The Straight-Through Gumbel-Softmax maintains the differentiability of the Gumbel-Softmax estimator while allowing for discrete sampling by taking different paths in the forward and backward pass. It uses argmax to get the one-hot sample z in the forward pass, but uses its continuous approximationz in the backward pass. While ST estimators are biased, they have been shown to work well in latent tree learning (Choi et al., 2018) and semisupervised machine translation (Niu et al., 2019).

Soft Aligned Maximum Likelihood
The soft aligned maximum likelihood (SAML) is defined as the probability that the reference can be aligned with the sampled output using a soft alignment predicted by the model: where T is the length of the reference sequence, T is the length of the sampled sequence, a tj is the predicted soft alignment between the reference word y t and sampled prefixỹ <j .
Training with the SAML objective consists in maximizing: The conditional probability of the next word p(y t |ỹ <j , x; θ) is computed as follows: where W and b are model parameters.h j is the hidden representation at step j conditioned on the  source sequence x and the preceding wordsỹ <j sampled from the model distribution using differentiable sampling: We compute the soft alignment a tj between y t andỹ <j based on the model's hidden states: where e yt is the embedding of the reference word y t . The score function captures the similarity between the hidden stateh j and the embedding e yt . We use the dot product here as it does not introduce additional parameters: score(h, e) = h e (11) Figure 1 illustrates how the resulting objective differs from scheduled sampling: (1) it is computed over sampled sequences as opposed to sequences that contain a mixture of sampled and reference words, and (2) each reference word is softaligned to the sampled sequence.

Experiments
Data We evaluate our approach on IWSLT 2014 German-English (de-en) as prior work (Goyal et al., 2017), as well as two additional tasks: IWSLT 2014 English-German (en-de) and IWSLT  Table 2: BLEU scores of our approach (SAML) and three baselines including the maximum likelihood (ML) baseline, scheduled sampling (SS), and differentiable scheduled sampling (DSS). The Anneal column indicates whether the sampling rate is annealed.
For each task, we report the mean and standard deviation over 5 runs with different random seeds. SAML achieves the best BLEU scores and is simpler to train than SS and DSS, as it requires no annealing schedule.
2015 Vietnamese-English (vi-en). For de-en and en-de, we follow the preprocessing steps in Ranzato et al. (2015). For vi-en, we use the data preprocessed by Luong and Manning (2015), with test2012 for validation and test2013 for testing. Table 1 summarizes the data statistics.
Setup Our translation models are attentional RNNs (Bahdanau et al., 2015) built on Sockeye (Hieber et al., 2017). We use bi-directional LSTM encoder and single-layer LSTM decoder with 256 hidden units, embeddings of size 256, and multilayer perceptron attention with a layer size of 256. We apply layer normalization (Ba et al., 2016) and label smoothing (0.1). We add dropout to embeddings (0.1) and decoder hidden states (0.2). For ST Gumbel-Softmax, we use temperature γ = 1 and noise scale β = 0.5. The decoding beam size is 5 unless stated otherwise. We train the models using the Adam optimizer (Kingma and Ba, 2015) with a batch size of 1024 words. We checkpoint models every 1000 updates. The initial learning rate is 0.0002, and it is reduced by 30% after 4 checkpoints without validation perplexity improvement. Training stops after 12 checkpoints without improvement. For training efficiency, we first pre-train a baseline model for each task using only J M L and fine-tune it using different approaches. In the fine-tuning phase, we inherit all settings except that we initialize the learning rate to 0.00002 and set the minimum number of checkpoints before early stopping to 24. We fine-tune each randomly seeded model independently.
Baselines We compare our model against three baselines: (1) a standard baseline trained with the ML objective, and models fine-tuned with (2) scheduled sampling (SS)  and (3) differentiable scheduled sampling (DSS) (Goyal et al., 2017). In SS and DSS, the probability of using reference words s is annealed using inverse sigmoid decay : s = k/(k + exp(i/k)) at the i-th checkpoint with k = 10.
Results Table 2 shows that the SAML improves over the ML baseline by +0.5 BLEU on de-en, +0.7 BLEU on en-de, and +1.0 BLEU on vi-en task. In addition, SAML consistently improves over both the scheduled sampling and differentiable scheduled sampling on all tasks. All improvements are significant with p < 0.002. Interestingly, differentiable scheduled sampling performs no better than scheduled sampling in our experiments, unlike in Goyal et al. (2017). Unlike scheduled sampling, our approach does not require an annealing schedule, and it is therefore simpler to train. We verify that the annealing schedule is needed in scheduled sampling by training a contrastive model with the same objective as scheduled sampling, but without annealing schedule (Table 2). We set the sampling rate to 0.5. The contrastive model hurts BLEU scores by at least 4.0 points compared to both the ML baseline and models fine-tuned with scheduled sampling, confirming that scheduled sampling needs the annealing schedule to work well.
We further examine the performance gain of different approaches over the baseline with varying beam sizes (Figure 2). Our approach yields larger BLEU improvements when decoding with greedy search and smaller beams, while there is no clear pattern for scheduled sampling models. These results support the hypothesis that our approach mitigates exposure bias, as it yields bigger improvements in settings where systems have fewer opportunities to recover from early errors. gorithms require an expert policy, which produces the best next token given any model predicted prefix, and assume that policy can be efficiently computed from the reference. However, for structured prediction tasks such as machine translation with large vocabulary and complex loss functions, it is intractable to find the best next token given any prefix. For time series modeling, the Data as Demonstrator algorithm (Venkatraman et al., 2015) derives the expert policy directly from the reference sequences which are aligned with the sampled sequences at each time step. Scheduled sampling algorithms Goyal et al., 2017) use the same strategy to train neural sequence-to-sequence models for a broader range of language generation tasks, even though the time alignment between reference and sampled sequences does not hold. Leblond et al. (2018) proposed to complete a predicted prefix with all possible reference suffixes and picking the reference suffix that yields the highest BLEU-1 score. However, they found that this approach performs well only when the prefix is close to the reference. Reinforcement learning (RL) algorithms (Bahdanau et al., 2016;Sutton and Barto, 2018;Van Hasselt et al., 2016) address exposure bias by directly optimizing a sentence-level reward for the model generated sequences. Evaluation metrics such as BLEU can be used as rewards, but they are discontinuous and hard to optimize. Techniques such as policy gradient (Williams, 1992) and actor-critic (Sutton and Barto, 2018;Degris et al., 2012) are thus required to find an unbiased estimation of the gradient to optimize the model. Due to the high variance of the gradient estimation, training with RL can be slow and unstable (Henderson et al., 2018;Wu et al., 2018). Recent alternatives use data augmentation to incorporate the sentence-level reward into the training objective more efficiently (Norouzi et al., 2016).

Related Work
Finally, our SAML loss shares the idea of flexible reference word order with the bag-of-word loss introduced by Ma et al. (2018) to improve source coverage. However, their loss is computed with teacher forcing and therefore does not address exposure bias.

Conclusion
We introduced a differentiable sampling algorithm which exposes a sequence-to-sequence model to its own predictions during training and compares them to reference sequences flexibly to backpropagate reliable error signals. By soft aligning reference and sampled sequences, our approach consistently improves BLEU over maximum likelihood and scheduled sampling baselines on three IWSLT tasks, with larger improvements for greedy search and smaller beam sizes. Our approach is also simple to train, as it does not require any sampling schedule.