Integrating Crowdsourcing and Active Learning for Classification of
  Work-Life Events from Tweets
  Social media, especially Twitter, is being increasingly used for research
with predictive analytics. In social media studies, natural language processing
(NLP) techniques are used in conjunction with expert-based, manual and
qualitative analyses. However, social media data are unstructured and must
undergo complex manipulation for research use. The manual annotation is the
most resource and time-consuming process that multiple expert raters have to
reach consensus on every item, but is essential to create gold-standard
datasets for training NLP-based machine learning classifiers. To reduce the
burden of the manual annotation, yet maintaining its reliability, we devised a
crowdsourcing pipeline combined with active learning strategies. We
demonstrated its effectiveness through a case study that identifies job loss
events from individual tweets. We used Amazon Mechanical Turk platform to
recruit annotators from the Internet and designed a number of quality control
measures to assure annotation accuracy. We evaluated 4 different active
learning strategies (i.e., least confident, entropy, vote entropy, and
Kullback-Leibler divergence). The active learning strategies aim at reducing
the number of tweets needed to reach a desired performance of automated
classification. Results show that crowdsourcing is useful to create
high-quality annotations and active learning helps in reducing the number of
required tweets, although there was no substantial difference among the
strategies tested.
