Mark-Evaluate: Assessing Language Generation using Population Estimation
  Methods
  We propose a family of metrics to assess language generation derived from
population estimation methods widely used in ecology. More specifically, we use
mark-recapture and maximum-likelihood methods that have been applied over the
past several decades to estimate the size of closed populations in the wild. We
propose three novel metrics: ME$_\text{Petersen}$ and ME$_\text{CAPTURE}$,
which retrieve a single-valued assessment, and ME$_\text{Schnabel}$ which
returns a double-valued metric to assess the evaluation set in terms of quality
and diversity, separately. In synthetic experiments, our family of methods is
sensitive to drops in quality and diversity. Moreover, our methods show a
higher correlation to human evaluation than existing metrics on several
challenging tasks, namely unconditional language generation, machine
translation, and text summarization.
