Dynamic Programming Encoding for Subword Segmentation in Neural Machine
  Translation
  This paper introduces Dynamic Programming Encoding (DPE), a new segmentation
algorithm for tokenizing sentences into subword units. We view the subword
segmentation of output sentences as a latent variable that should be
marginalized out for learning and inference. A mixed character-subword
transformer is proposed, which enables exact log marginal likelihood estimation
and exact MAP inference to find target segmentations with maximum posterior
probability. DPE uses a lightweight mixed character-subword transformer as a
means of pre-processing parallel data to segment output sentences using dynamic
programming. Empirical results on machine translation suggest that DPE is
effective for segmenting output sentences and can be combined with BPE dropout
for stochastic segmentation of source sentences. DPE achieves an average
improvement of 0.9 BLEU over BPE (Sennrich et al., 2016) and an average
improvement of 0.55 BLEU over BPE dropout (Provilkov et al., 2019) on several
WMT datasets including English <=> (German, Romanian, Estonian, Finnish,
Hungarian).
