Feedback-Based Self-Learning in Large-Scale Conversational AI Agents
  Today, most large-scale conversational AI agents (e.g. Alexa, Siri, or Google
Assistant) are built using manually annotated data to train the different
components of the system. Typically, the accuracy of the ML models in these
components are improved by manually transcribing and annotating data. As the
scope of these systems increase to cover more scenarios and domains, manual
annotation to improve the accuracy of these components becomes prohibitively
costly and time consuming. In this paper, we propose a system that leverages
user-system interaction feedback signals to automate learning without any
manual annotation. Users here tend to modify a previous query in hopes of
fixing an error in the previous turn to get the right results. These
reformulations, which are often preceded by defective experiences caused by
errors in ASR, NLU, ER or the application. In some cases, users may not
properly formulate their requests (e.g. providing partial title of a song), but
gleaning across a wider pool of users and sessions reveals the underlying
recurrent patterns. Our proposed self-learning system automatically detects the
errors, generate reformulations and deploys fixes to the runtime system to
correct different types of errors occurring in different components of the
system. In particular, we propose leveraging an absorbing Markov Chain model as
a collaborative filtering mechanism in a novel attempt to mine these patterns.
We show that our approach is highly scalable, and able to learn reformulations
that reduce Alexa-user errors by pooling anonymized data across millions of
customers. The proposed self-learning system achieves a win/loss ratio of 11.8
and effectively reduces the defect rate by more than 30% on utterance level
reformulations in our production A/B tests. To the best of our knowledge, this
is the first self-learning large-scale conversational AI system in production.
