Leveraging User Engagement Signals For Entity Labeling in a Virtual
  Assistant
  Personal assistant AI systems such as Siri, Cortana, and Alexa have become
widely used as a means to accomplish tasks through natural language commands.
However, components in these systems generally rely on supervised machine
learning algorithms that require large amounts of hand-annotated training data,
which is expensive and time consuming to collect. The ability to incorporate
unsupervised, weakly supervised, or distantly supervised data holds significant
promise in overcoming this bottleneck. In this paper, we describe a framework
that leverages user engagement signals (user behaviors that demonstrate a
positive or negative response to content) to automatically create granular
entity labels for training data augmentation. Strategies such as multi-task
learning and validation using an external knowledge base are employed to
incorporate the engagement annotated data and to boost the model's accuracy on
a sequence labeling task. Our results show that learning from data
automatically labeled by user engagement signals achieves significant accuracy
gains in a production deep learning system, when measured on both the sequence
labeling task as well as on user facing results produced by the system
end-to-end. We believe this is the first use of user engagement signals to help
generate training data for a sequence labeling task on a large scale, and can
be applied in practical settings to speed up new feature deployment when little
human annotated data is available.
