On the Evaluation of Dialogue Systems with Next Utterance Classification
  An open challenge in constructing dialogue systems is developing methods for
automatically learning dialogue strategies from large amounts of unlabelled
data. Recent work has proposed Next-Utterance-Classification (NUC) as a
surrogate task for building dialogue systems from text data. In this paper we
investigate the performance of humans on this task to validate the relevance of
NUC as a method of evaluation. Our results show three main findings: (1) humans
are able to correctly classify responses at a rate much better than chance,
thus confirming that the task is feasible, (2) human performance levels vary
across task domains (we consider 3 datasets) and expertise levels (novice vs
experts), thus showing that a range of performance is possible on this type of
task, (3) automated dialogue systems built using state-of-the-art machine
learning methods have similar performance to the human novices, but worse than
the experts, thus confirming the utility of this class of tasks for driving
further research in automated dialogue systems.
