Comparison-Based Framework for Psychophysics: Lab versus Crowdsourcing
  Traditionally, psychophysical experiments are conducted by repeated
measurements on a few well-trained participants under well-controlled
conditions, often resulting in, if done properly, high quality data. In recent
years, however, crowdsourcing platforms are becoming increasingly popular means
of data collection, measuring many participants at the potential cost of
obtaining data of worse quality. In this paper we study whether the use of
comparison-based (ordinal) data, combined with machine learning algorithms, can
boost the reliability of crowdsourcing studies for psychophysics, such that
they can achieve performance close to a lab experiment. To this end, we compare
three setups: simulations, a psychophysics lab experiment, and the same
experiment on Amazon Mechanical Turk. All these experiments are conducted in a
comparison-based setting where participants have to answer triplet questions of
the form "is object x closer to y or to z?". We then use machine learning to
solve the triplet prediction problem: given a subset of triplet questions with
corresponding answers, we predict the answer to the remaining questions.
Considering the limitations and noise on MTurk, we find that the accuracy of
triplet prediction is surprisingly close---but not equal---to our lab study.
