BLEU might be Guilty but References are not Innocent
  The quality of automatic metrics for machine translation has been
increasingly called into question, especially for high-quality systems. This
paper demonstrates that, while choice of metric is important, the nature of the
references is also critical. We study different methods to collect references
and compare their value in automated evaluation by reporting correlation with
human evaluation for a variety of systems and metrics. Motivated by the finding
that typical references exhibit poor diversity, concentrating around
translationese language, we develop a paraphrasing task for linguists to
perform on existing reference translations, which counteracts this bias. Our
method yields higher correlation with human judgment not only for the
submissions of WMT 2019 English to German, but also for Back-translation and
APE augmented MT output, which have been shown to have low correlation with
automatic metrics using standard references. We demonstrate that our
methodology improves correlation with all modern evaluation metrics we look at,
including embedding-based methods. To complete this picture, we reveal that
multi-reference BLEU does not improve the correlation for high quality output,
and present an alternative multi-reference formulation that is more effective.
