I like fish, especially dolphins: Addressing Contradictions in Dialogue
  Modeling
  To quantify how well natural language understanding models can capture
consistency in a general conversation, we introduce the DialoguE COntradiction
DEtection task (DECODE) and a new conversational dataset containing both
human-human and human-bot contradictory dialogues. We then compare a structured
utterance-based approach of using pre-trained Transformer models for
contradiction detection with the typical unstructured approach. Results reveal
that: (i) our newly collected dataset is notably more effective at providing
supervision for the dialogue contradiction detection task than existing NLI
data including those aimed to cover the dialogue domain; (ii) the structured
utterance-based approach is more robust and transferable on both analysis and
out-of-distribution dialogues than its unstructured counterpart. We also show
that our best contradiction detection model correlates well with human
judgments and further provide evidence for its usage in both automatically
evaluating and improving the consistency of state-of-the-art generative
chatbots.
