Learning to Hire Teams
  Crowdsourcing and human computation has been employed in increasingly
sophisticated projects that require the solution of a heterogeneous set of
tasks. We explore the challenge of building or hiring an effective team, for
performing tasks required for such projects on an ongoing basis, from an
available pool of applicants or workers who have bid for the tasks. The
recruiter needs to learn workers' skills and expertise by performing online
tests and interviews, and would like to minimize the amount of budget or time
spent in this process before committing to hiring the team. How can one
optimally spend budget to learn the expertise of workers as part of recruiting
a team? How can one exploit the similarities among tasks as well as underlying
social ties or commonalities among the workers for faster learning? We tackle
these decision-theoretic challenges by casting them as an instance of online
learning for best action selection. We present algorithms with PAC bounds on
the required budget to hire a near-optimal team with high confidence.
Furthermore, we consider an embedding of the tasks and workers in an underlying
graph that may arise from task similarities or social ties, and that can
provide additional side-observations for faster learning. We then quantify the
improvement in the bounds that we can achieve depending on the characteristic
properties of this graph structure. We evaluate our methodology on simulated
problem instances as well as on real-world crowdsourcing data collected from
the oDesk platform. Our methodology and results present an interesting
direction of research to tackle the challenges faced by a recruiter for
contract-based crowdsourcing.
