Active Sampling for Pairwise Comparisons via Approximate Message Passing
  and Information Gain Maximization
  Pairwise comparison data arise in many domains with subjective assessment
experiments, for example in image and video quality assessment. In these
experiments observers are asked to express a preference between two conditions.
However, many pairwise comparison protocols require a large number of
comparisons to infer accurate scores, which may be unfeasible when each
comparison is time-consuming (e.g. videos) or expensive (e.g. medical imaging).
This motivates the use of an active sampling algorithm that chooses only the
most informative pairs for comparison. In this paper we propose ASAP, an active
sampling algorithm based on approximate message passing and expected
information gain maximization. Unlike most existing methods, which rely on
partial updates of the posterior distribution, we are able to perform full
updates and therefore much improve the accuracy of the inferred scores. The
algorithm relies on three techniques for reducing computational cost: inference
based on approximate message passing, selective evaluations of the information
gain, and selecting pairs in a batch that forms a minimum spanning tree of the
inverse of information gain. We demonstrate, with real and synthetic data, that
ASAP offers the highest accuracy of inferred scores compared to the existing
methods. We also provide an open-source GPU implementation of ASAP for
large-scale experiments.
