Consistent Position Bias Estimation without Online Interventions for
  Learning-to-Rank
  Presentation bias is one of the key challenges when learning from implicit
feedback in search engines, as it confounds the relevance signal with
uninformative signals due to position in the ranking, saliency, and other
presentation factors. While it was recently shown how counterfactual
learning-to-rank (LTR) approaches \cite{Joachims/etal/17a} can provably
overcome presentation bias if observation propensities are known, it remains to
show how to accurately estimate these propensities. In this paper, we propose
the first method for producing consistent propensity estimates without manual
relevance judgments, disruptive interventions, or restrictive relevance
modeling assumptions. We merely require that we have implicit feedback data
from multiple different ranking functions. Furthermore, we argue that our
estimation technique applies to an extended class of Contextual Position-Based
Propensity Models, where propensities not only depend on position but also on
observable features of the query and document. Initial simulation studies
confirm that the approach is scalable, accurate, and robust.
