Active Annotation: bootstrapping annotation lexicon and guidelines for
  supervised NLU learning
  Natural Language Understanding (NLU) models are typically trained in a
supervised learning framework. In the case of intent classification, the
predicted labels are predefined and based on the designed annotation schema
while the labelling process is based on a laborious task where annotators
manually inspect each utterance and assign the corresponding label. We propose
an Active Annotation (AA) approach where we combine an unsupervised learning
method in the embedding space, a human-in-the-loop verification process, and
linguistic insights to create lexicons that can be open categories and adapted
over time. In particular, annotators define the y-label space on-the-fly during
the annotation using an iterative process and without the need for prior
knowledge about the input data. We evaluate the proposed annotation paradigm in
a real use-case NLU scenario. Results show that our Active Annotation paradigm
achieves accurate and higher quality training data, with an annotation speed of
an order of magnitude higher with respect to the traditional human-only driven
baseline annotation methodology.
