On Long-Tailed Phenomena in Neural Machine Translation
  State-of-the-art Neural Machine Translation (NMT) models struggle with
generating low-frequency tokens, tackling which remains a major challenge. The
analysis of long-tailed phenomena in the context of structured prediction tasks
is further hindered by the added complexities of search during inference. In
this work, we quantitatively characterize such long-tailed phenomena at two
levels of abstraction, namely, token classification and sequence generation. We
propose a new loss function, the Anti-Focal loss, to better adapt model
training to the structural dependencies of conditional text generation by
incorporating the inductive biases of beam search in the training process. We
show the efficacy of the proposed technique on a number of Machine Translation
(MT) datasets, demonstrating that it leads to significant gains over
cross-entropy across different language pairs, especially on the generation of
low-frequency words. We have released the code to reproduce our results.
