Error Rate Bounds and Iterative Weighted Majority Voting for
  Crowdsourcing
  Crowdsourcing has become an effective and popular tool for human-powered
computation to label large datasets. Since the workers can be unreliable, it is
common in crowdsourcing to assign multiple workers to one task, and to
aggregate the labels in order to obtain results of high quality. In this paper,
we provide finite-sample exponential bounds on the error rate (in probability
and in expectation) of general aggregation rules under the Dawid-Skene
crowdsourcing model. The bounds are derived for multi-class labeling, and can
be used to analyze many aggregation methods, including majority voting,
weighted majority voting and the oracle Maximum A Posteriori (MAP) rule. We
show that the oracle MAP rule approximately optimizes our upper bound on the
mean error rate of weighted majority voting in certain setting. We propose an
iterative weighted majority voting (IWMV) method that optimizes the error rate
bound and approximates the oracle MAP rule. Its one step version has a provable
theoretical guarantee on the error rate. The IWMV method is intuitive and
computationally simple. Experimental results on simulated and real data show
that IWMV performs at least on par with the state-of-the-art methods, and it
has a much lower computational cost (around one hundred times faster) than the
state-of-the-art methods.
