Analysis of Multivariate Scoring Functions for Automatic Unbiased
  Learning to Rank
  Leveraging biased click data for optimizing learning to rank systems has been
a popular approach in information retrieval. Because click data is often noisy
and biased, a variety of methods have been proposed to construct unbiased
learning to rank (ULTR) algorithms for the learning of unbiased ranking models.
Among them, automatic unbiased learning to rank (AutoULTR) algorithms that
jointly learn user bias models (i.e., propensity models) with unbiased rankers
have received a lot of attention due to their superior performance and low
deployment cost in practice. Despite their differences in theories and
algorithm design, existing studies on ULTR usually use uni-variate ranking
functions to score each document or result independently. On the other hand,
recent advances in context-aware learning-to-rank models have shown that
multivariate scoring functions, which read multiple documents together and
predict their ranking scores jointly, are more powerful than uni-variate
ranking functions in ranking tasks with human-annotated relevance labels.
Whether such superior performance would hold in ULTR with noisy data, however,
is mostly unknown. In this paper, we investigate existing multivariate scoring
functions and AutoULTR algorithms in theory and prove that permutation
invariance is a crucial factor that determines whether a context-aware
learning-to-rank model could be applied to existing AutoULTR framework. Our
experiments with synthetic clicks on two large-scale benchmark datasets show
that AutoULTR models with permutation-invariant multivariate scoring functions
significantly outperform those with uni-variate scoring functions and
permutation-variant multivariate scoring functions.
