Modeling Multiple Annotator Expertise in the Semi-Supervised Learning
  Scenario
  Learning algorithms normally assume that there is at most one annotation or
label per data point. However, in some scenarios, such as medical diagnosis and
on-line collaboration,multiple annotations may be available. In either case,
obtaining labels for data points can be expensive and time-consuming (in some
circumstances ground-truth may not exist). Semi-supervised learning approaches
have shown that utilizing the unlabeled data is often beneficial in these
cases. This paper presents a probabilistic semi-supervised model and algorithm
that allows for learning from both unlabeled and labeled data in the presence
of multiple annotators. We assume that it is known what annotator labeled which
data points. The proposed approach produces annotator models that allow us to
provide (1) estimates of the true label and (2) annotator variable expertise
for both labeled and unlabeled data. We provide numerical comparisons under
various scenarios and with respect to standard semi-supervised learning.
Experiments showed that the presented approach provides clear advantages over
multi-annotator methods that do not use the unlabeled data and over methods
that do not use multi-labeler information.
