Distributional Ground Truth: Non-Redundant Crowdsourcing Data Quality
  Control in UI Labeling Tasks
  HCI increasingly employs Machine Learning and Image Recognition, in
particular for visual analysis of user interfaces (UIs). A popular way for
obtaining human-labeled training data is Crowdsourcing, typically using the
quality control methods ground truth and majority consensus, which necessitate
redundancy in the outcome. In our paper we propose a non-redundant method for
prediction of crowdworkers' output quality in web UI labeling tasks, based on
homogeneity of distributions assessed with two-sample Kolmogorov-Smirnov test.
Using a dataset of about 500 screenshots with over 74,000 UI elements located
and classified by 11 trusted labelers and 298 Amazon Mechanical Turk
crowdworkers, we demonstrate the advantage of our approach over the baseline
model based on mean Time-on-Task. Exploring different dataset partitions, we
show that with the trusted set size of 17-27% UIs our "distributional ground
truth" model can achieve R2s of over 0.8 and help to obviate the ancillary work
effort and expenses.
