Reward Optimization for Neural Machine Translation with Learned Metrics
  Neural machine translation (NMT) models are conventionally trained with
token-level negative log-likelihood (NLL), which does not guarantee that the
generated translations will be optimized for a selected sequence-level
evaluation metric. Multiple approaches are proposed to train NMT with BLEU as
the reward, in order to directly improve the metric. However, it was reported
that the gain in BLEU does not translate to real quality improvement, limiting
the application in industry. Recently, it became clear to the community that
BLEU has a low correlation with human judgment when dealing with
state-of-the-art models. This leads to the emerging of model-based evaluation
metrics. These new metrics are shown to have a much higher human correlation.
In this paper, we investigate whether it is beneficial to optimize NMT models
with the state-of-the-art model-based metric, BLEURT. We propose a
contrastive-margin loss for fast and stable reward optimization suitable for
large NMT models. In experiments, we perform automatic and human evaluations to
compare models trained with smoothed BLEU and BLEURT to the baseline models.
Results show that the reward optimization with BLEURT is able to increase the
metric scores by a large margin, in contrast to limited gain when training with
smoothed BLEU. The human evaluation shows that models trained with BLEURT
improve adequacy and coverage of translations. Code is available via
https://github.com/naver-ai/MetricMT.
