Provably Good Batch Reinforcement Learning Without Great Exploration
  Batch reinforcement learning (RL) is important to apply RL algorithms to many
high stakes tasks. Doing batch RL in a way that yields a reliable new policy in
large domains is challenging: a new decision policy may visit states and
actions outside the support of the batch data, and function approximation and
optimization with limited samples can further increase the potential of
learning policies with overly optimistic estimates of their future performance.
Recent algorithms have shown promise but can still be overly optimistic in
their expected outcomes. Theoretical work that provides strong guarantees on
the performance of the output policy relies on a strong concentrability
assumption, that makes it unsuitable for cases where the ratio between
state-action distributions of behavior policy and some candidate policies is
large. This is because in the traditional analysis, the error bound scales up
with this ratio. We show that a small modification to Bellman optimality and
evaluation back-up to take a more conservative update can have much stronger
guarantees. In certain settings, they can find the approximately best policy
within the state-action space explored by the batch data, without requiring a
priori assumptions of concentrability. We highlight the necessity of our
conservative update and the limitations of previous algorithms and analyses by
illustrative MDP examples, and demonstrate an empirical comparison of our
algorithm and other state-of-the-art batch RL baselines in standard benchmarks.
