Bellman Residual Orthogonalization for Offline Reinforcement Learning
  We introduce a new reinforcement learning principle that approximates the
Bellman equations by enforcing their validity only along an user-defined space
of test functions. Focusing on applications to model-free offline RL with
function approximation, we exploit this principle to derive confidence
intervals for off-policy evaluation, as well as to optimize over policies
within a prescribed policy class. We prove an oracle inequality on our policy
optimization procedure in terms of a trade-off between the value and
uncertainty of an arbitrary comparator policy. Different choices of test
function spaces allow us to tackle different problems within a common
framework. We characterize the loss of efficiency in moving from on-policy to
off-policy data using our procedures, and establish connections to
concentrability coefficients studied in past work. We examine in depth the
implementation of our methods with linear function approximation, and provide
theoretical guarantees with polynomial-time implementations even when Bellman
closure does not hold.
