Confounding-Robust Policy Evaluation in Infinite-Horizon Reinforcement
  Learning
  Off-policy evaluation of sequential decision policies from observational data
is necessary in applications of batch reinforcement learning such as education
and healthcare. In such settings, however, unobserved variables confound
observed actions, rendering exact evaluation of new policies impossible, i.e.,
unidentifiable. We develop a robust approach that estimates sharp bounds on the
(unidentifiable) value of a given policy in an infinite-horizon problem given
data from another policy with unobserved confounding, subject to a sensitivity
model. We consider stationary or baseline unobserved confounding and compute
bounds by optimizing over the set of all stationary state-occupancy ratios that
agree with a new partially identified estimating equation and the sensitivity
model. We prove convergence to the sharp bounds as we collect more confounded
data. Although checking set membership is a linear program, the support
function is given by a difficult nonconvex optimization problem. We develop
approximations based on nonconvex projected gradient descent and demonstrate
the resulting bounds empirically.
