Benchmarks for Deep Off-Policy Evaluation
  Off-policy evaluation (OPE) holds the promise of being able to leverage
large, offline datasets for both evaluating and selecting complex policies for
decision making. The ability to learn offline is particularly important in many
real-world domains, such as in healthcare, recommender systems, or robotics,
where online data collection is an expensive and potentially dangerous process.
Being able to accurately evaluate and select high-performing policies without
requiring online interaction could yield significant benefits in safety, time,
and cost for these applications. While many OPE methods have been proposed in
recent years, comparing results between papers is difficult because currently
there is a lack of a comprehensive and unified benchmark, and measuring
algorithmic progress has been challenging due to the lack of difficult
evaluation tasks. In order to address this gap, we present a collection of
policies that in conjunction with existing offline datasets can be used for
benchmarking off-policy evaluation. Our tasks include a range of challenging
high-dimensional continuous control problems, with wide selections of datasets
and policies for performing policy selection. The goal of our benchmark is to
provide a standardized measure of progress that is motivated from a set of
principles designed to challenge and test the limits of existing OPE methods.
We perform an evaluation of state-of-the-art algorithms and provide open-source
access to our data and code to foster future research in this area.
