Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation
  Practices for Generated Text
  Evaluation practices in natural language generation (NLG) have many known
flaws, but improved evaluation approaches are rarely widely adopted. This issue
has become more urgent, since neural NLG models have improved to the point
where they can often no longer be distinguished based on the surface-level
features that older metrics rely on. This paper surveys the issues with human
and automatic model evaluations and with commonly used datasets in NLG that
have been pointed out over the past 20 years. We summarize, categorize, and
discuss how researchers have been addressing these issues and what their
findings mean for the current state of model evaluations. Building on those
insights, we lay out a long-term vision for NLG evaluation and propose concrete
steps for researchers to improve their evaluation processes. Finally, we
analyze 66 NLG papers from recent NLP conferences in how well they already
follow these suggestions and identify which areas require more drastic changes
to the status quo.
