Accountable Off-Policy Evaluation With Kernel Bellman Statistics
  We consider off-policy evaluation (OPE), which evaluates the performance of a
new policy from observed data collected from previous experiments, without
requiring the execution of the new policy. This finds important applications in
areas with high execution cost or safety concerns, such as medical diagnosis,
recommendation systems and robotics. In practice, due to the limited
information from off-policy data, it is highly desirable to construct rigorous
confidence intervals, not just point estimation, for the policy performance. In
this work, we propose a new variational framework which reduces the problem of
calculating tight confidence bounds in OPE into an optimization problem on a
feasible set that catches the true state-action value function with high
probability. The feasible set is constructed by leveraging statistical
properties of a recently proposed kernel Bellman loss (Feng et al., 2019). We
design an efficient computational approach for calculating our bounds, and
extend it to perform post-hoc diagnosis and correction for existing estimators.
Empirical results show that our method yields tight confidence intervals in
different settings.
