Sequential Multi-Class Labeling in Crowdsourcing
  We consider a crowdsourcing platform where workers' responses to questions
posed by a crowdsourcer are used to determine the hidden state of a multi-class
labeling problem. As workers may be unreliable, we propose to perform
sequential questioning in which the questions posed to the workers are designed
based on previous questions and answers. We propose a Partially-Observable
Markov Decision Process (POMDP) framework to determine the best questioning
strategy, subject to the crowdsourcer's budget constraint. As this POMDP
formulation is in general intractable, we develop a suboptimal approach based
on a $q$-ary Ulam-R\'enyi game. We also propose a sampling heuristic, which can
be used in tandem with standard POMDP solvers, using our Ulam-R\'enyi strategy.
We demonstrate through simulations that our approaches outperform a
non-sequential strategy based on error correction coding and which does not
utilize workers' previous responses.
