Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible
  Off-Policy Evaluation
  Off-policy evaluation (OPE) aims to estimate the performance of hypothetical
policies using data generated by a different policy. Because of its huge
potential impact in practice, there has been growing research interest in this
field. There is, however, no real-world public dataset that enables the
evaluation of OPE, making its experimental studies unrealistic and
irreproducible. With the goal of enabling realistic and reproducible OPE
research, we present Open Bandit Dataset, a public logged bandit dataset
collected on a large-scale fashion e-commerce platform, ZOZOTOWN. Our dataset
is unique in that it contains a set of multiple logged bandit datasets
collected by running different policies on the same platform. This enables
experimental comparisons of different OPE estimators for the first time. We
also develop Python software called Open Bandit Pipeline to streamline and
standardize the implementation of batch bandit algorithms and OPE. Our open
data and software will contribute to fair and transparent OPE research and help
the community identify fruitful research directions. We provide extensive
benchmark experiments of existing OPE estimators using our dataset and
software. The results open up essential challenges and new avenues for future
OPE research.
