Modeling Human Annotation Errors to Design Bias-Aware Systems for Social
  Stream Processing
  High-quality human annotations are necessary to create effective machine
learning systems for social media. Low-quality human annotations indirectly
contribute to the creation of inaccurate or biased learning systems. We show
that human annotation quality is dependent on the ordering of instances shown
to annotators (referred as 'annotation schedule'), and can be improved by local
changes in the instance ordering provided to the annotators, yielding a more
accurate annotation of the data stream for efficient real-time social media
analytics. We propose an error-mitigating active learning algorithm that is
robust with respect to some cases of human errors when deciding an annotation
schedule. We validate the human error model and evaluate the proposed algorithm
against strong baselines by experimenting on classification tasks of relevant
social media posts during crises. According to these experiments, considering
the order in which data instances are presented to human annotators leads to
both an increase in accuracy for machine learning and awareness toward some
potential biases in human learning that may affect the automated classifier.
