Error Rate Bounds in Crowdsourcing Models
  Crowdsourcing is an effective tool for human-powered computation on many
tasks challenging for computers. In this paper, we provide finite-sample
exponential bounds on the error rate (in probability and in expectation) of
hyperplane binary labeling rules under the Dawid-Skene crowdsourcing model. The
bounds can be applied to analyze many common prediction methods, including the
majority voting and weighted majority voting. These bound results could be
useful for controlling the error rate and designing better algorithms. We show
that the oracle Maximum A Posterior (MAP) rule approximately optimizes our
upper bound on the mean error rate for any hyperplane binary labeling rule, and
propose a simple data-driven weighted majority voting (WMV) rule (called
one-step WMV) that attempts to approximate the oracle MAP and has a provable
theoretical guarantee on the error rate. Moreover, we use simulated and real
data to demonstrate that the data-driven EM-MAP rule is a good approximation to
the oracle MAP rule, and to demonstrate that the mean error rate of the
data-driven EM-MAP rule is also bounded by the mean error rate bound of the
oracle MAP rule with estimated parameters plugging into the bound.
