Two-Sample Testing on Ranked Preference Data and the Role of Modeling
  Assumptions
  A number of applications require two-sample testing on ranked preference
data. For instance, in crowdsourcing, there is a long-standing question of
whether pairwise comparison data provided by people is distributed similar to
ratings-converted-to-comparisons. Other examples include sports data analysis
and peer grading. In this paper, we design two-sample tests for pairwise
comparison data and ranking data. For our two-sample test for pairwise
comparison data, we establish an upper bound on the sample complexity required
to correctly distinguish between the distributions of the two sets of samples.
Our test requires essentially no assumptions on the distributions. We then
prove complementary lower bounds showing that our results are tight (in the
minimax sense) up to constant factors. We investigate the role of modeling
assumptions by proving lower bounds for a range of pairwise comparison models
(WST, MST,SST, parameter-based such as BTL and Thurstone). We also provide
testing algorithms and associated sample complexity bounds for the problem of
two-sample testing with partial (or total) ranking data.Furthermore, we
empirically evaluate our results via extensive simulations as well as two
real-world datasets consisting of pairwise comparisons. By applying our
two-sample test on real-world pairwise comparison data, we conclude that
ratings and rankings provided by people are indeed distributed differently. On
the other hand, our test recognizes no significant difference in the relative
performance of European football teams across two seasons. Finally, we apply
our two-sample test on a real-world partial and total ranking dataset and find
a statistically significant difference in Sushi preferences across demographic
divisions based on gender, age and region of residence.
