Parsimonious Mixed-Effects HodgeRank for Crowdsourced Preference
  Aggregation
  In crowdsourced preference aggregation, it is often assumed that all the
annotators are subject to a common preference or utility function which
generates their comparison behaviors in experiments. However, in reality
annotators are subject to variations due to multi-criteria, abnormal, or a
mixture of such behaviors. In this paper, we propose a parsimonious
mixed-effects model based on HodgeRank, which takes into account both the fixed
effect that the majority of annotators follows a common linear utility model,
and the random effect that a small subset of annotators might deviate from the
common significantly and exhibits strongly personalized preferences. HodgeRank
has been successfully applied to subjective quality evaluation of multimedia
and resolves pairwise crowdsourced ranking data into a global consensus ranking
and cyclic conflicts of interests. As an extension, our proposed methodology
further explores the conflicts of interests through the random effect in
annotator specific variations. The key algorithm in this paper establishes a
dynamic path from the common utility to individual variations, with different
levels of parsimony or sparsity on personalization, based on newly developed
Linearized Bregman Algorithms with Inverse Scale Space method. Finally the
validity of the methodology are supported by experiments with both simulated
examples and three real-world crowdsourcing datasets, which shows that our
proposed method exhibits better performance (i.e. smaller test error) compared
with HodgeRank due to its parsimonious property.
