TagRuler: Interactive Tool for Span-Level Data Programming by
  Demonstration
  Despite rapid developments in the field of machine learning research,
collecting high-quality labels for supervised learning remains a bottleneck for
many applications. This difficulty is exacerbated by the fact that
state-of-the-art models for NLP tasks are becoming deeper and more complex,
often increasing the amount of training data required even for fine-tuning.
Weak supervision methods, including data programming, address this problem and
reduce the cost of label collection by using noisy label sources for
supervision. However, until recently, data programming was only accessible to
users who knew how to program. To bridge this gap, the Data Programming by
Demonstration framework was proposed to facilitate the automatic creation of
labeling functions based on a few examples labeled by a domain expert. This
framework has proven successful for generating high-accuracy labeling models
for document classification. In this work, we extend the DPBD framework to
span-level annotation tasks, arguably one of the most time-consuming NLP
labeling tasks. We built a novel tool, TagRuler, that makes it easy for
annotators to build span-level labeling functions without programming and
encourages them to explore trade-offs between different labeling models and
active learning strategies. We empirically demonstrated that an annotator could
achieve a higher F1 score using the proposed tool compared to manual labeling
for different span-level annotation tasks.
