ULTRA: An Unbiased Learning To Rank Algorithm Toolbox
  Learning to rank systems has become an important aspect of our daily life.
However, the implicit user feedback that is used to train many learning to rank
models is usually noisy and suffered from user bias (i.e., position bias).
Thus, obtaining an unbiased model using biased feedback has become an important
research field for IR. Existing studies on unbiased learning to rank (ULTR) can
be generalized into two families-algorithms that attain unbiasedness with
logged data, offline learning, and algorithms that achieve unbiasedness by
estimating unbiased parameters with real-time user interactions, namely online
learning. While there exist many algorithms from both families, there lacks a
unified way to compare and benchmark them. As a result, it can be challenging
for researchers to choose the right technique for their problems or for people
who are new to the field to learn and understand existing algorithms. To solve
this problem, we introduced ULTRA, which is a flexible, extensible, and easily
configure ULTR toolbox. Its key features include support for multiple ULTR
algorithms with configurable hyperparameters, a variety of built-in click
models that can be used separately to simulate clicks, different ranking model
architecture and evaluation metrics, and simple learning to rank pipeline
creation. In this paper, we discuss the general framework of ULTR, briefly
describe the algorithms in ULTRA, detailed the structure, and pipeline of the
toolbox. We experimented on all the algorithms supported by ultra and showed
that the toolbox performance is reasonable. Our toolbox is an important
resource for researchers to conduct experiments on ULTR algorithms with
different configurations as well as testing their own algorithms with the
supported features.
