Multi-Label Active Learning from Crowds
  Multi-label active learning is a hot topic in reducing the label cost by
optimally choosing the most valuable instance to query its label from an
oracle. In this paper, we consider the poolbased multi-label active learning
under the crowdsourcing setting, where during the active query process, instead
of resorting to a high cost oracle for the ground-truth, multiple low cost
imperfect annotators with various expertise are available for labeling. To deal
with this problem, we propose the MAC (Multi-label Active learning from Crowds)
approach which incorporate the local influence of label correlations to build a
probabilistic model over the multi-label classifier and annotators. Based on
this model, we can estimate the labels for instances as well as the expertise
of each annotator. Then we propose the instance selection and annotator
selection criteria that consider the uncertainty/diversity of instances and the
reliability of annotators, such that the most reliable annotator will be
queried for the most valuable instances. Experimental results demonstrate the
effectiveness of the proposed approach.
