Semi-verified PAC Learning from the Crowd with Pairwise Comparisons
  We study the problem of crowdsourced PAC learning of threshold functions with
pairwise comparisons. This is a challenging problem and only recently have
query-efficient algorithms been established in the scenario where the majority
of the crowd are perfect. In this work, we investigate the significantly more
challenging case that the majority are incorrect, which in general renders
learning impossible. We show that under the semi-verified model of
Charikar~et~al.~(2017), where we have (limited) access to a trusted oracle who
always returns the correct annotation, it is possible to PAC learn the
underlying hypothesis class while drastically mitigating the labeling cost via
the more easily obtained comparison queries. Orthogonal to recent developments
in semi-verified or list-decodable learning that crucially rely on data
distributional assumptions, our PAC guarantee holds by exploring the wisdom of
the crowd.
