Learning an Unreferenced Metric for Online Dialogue Evaluation
  Evaluating the quality of a dialogue interaction between two agents is a
difficult task, especially in open-domain chit-chat style dialogue. There have
been recent efforts to develop automatic dialogue evaluation metrics, but most
of them do not generalize to unseen datasets and/or need a human-generated
reference response during inference, making it infeasible for online
evaluation. Here, we propose an unreferenced automated evaluation metric that
uses large pre-trained language models to extract latent representations of
utterances, and leverages the temporal transitions that exist between them. We
show that our model achieves higher correlation with human annotations in an
online setting, while not requiring true responses for comparison during
inference.
