A General Early-Stopping Module for Crowdsourced Ranking
  Crowdsourcing can be used to determine a total order for an object set (e.g.,
the top-10 NBA players) based on crowd opinions. This ranking problem is often
decomposed into a set of microtasks (e.g., pairwise comparisons). These
microtasks are passed to a large number of workers and their answers are
aggregated to infer the ranking. The number of microtasks depends on the budget
allocated for the problem. Intuitively, the higher the number of microtask
answers, the more accurate the ranking becomes. However, it is often hard to
decide the budget required for an accurate ranking. We study how a ranking
process can be terminated early, and yet achieve a high-quality ranking and
great savings in the budget. We use statistical tools to estimate the quality
of the ranking result at any stage of the crowdsourcing process and terminate
the process as soon as the desired quality is achieved. Our proposed
early-stopping module can be seamlessly integrated with most existing inference
algorithms and task assignment methods. We conduct extensive experiments and
show that our early-stopping module is better than other existing general
stopping criteria. We also implement a prototype system to demonstrate the
usability and effectiveness of our approach in practice.
