Estimation of Fair Ranking Metrics with Incomplete Judgments
  There is increasing attention to evaluating the fairness of search system
ranking decisions. These metrics often consider the membership of items to
particular groups, often identified using protected attributes such as gender
or ethnicity. To date, these metrics typically assume the availability and
completeness of protected attribute labels of items. However, the protected
attributes of individuals are rarely present, limiting the application of fair
ranking metrics in large scale systems. In order to address this problem, we
propose a sampling strategy and estimation technique for four fair ranking
metrics. We formulate a robust and unbiased estimator which can operate even
with very limited number of labeled items. We evaluate our approach using both
simulated and real world data. Our experimental results demonstrate that our
method can estimate this family of fair ranking metrics and provides a robust,
reliable alternative to exhaustive or random data annotation.
