Bellman-consistent Pessimism for Offline Reinforcement Learning
  The use of pessimism, when reasoning about datasets lacking exhaustive
exploration has recently gained prominence in offline reinforcement learning.
Despite the robustness it adds to the algorithm, overly pessimistic reasoning
can be equally damaging in precluding the discovery of good policies, which is
an issue for the popular bonus-based pessimism. In this paper, we introduce the
notion of Bellman-consistent pessimism for general function approximation:
instead of calculating a point-wise lower bound for the value function, we
implement pessimism at the initial state over the set of functions consistent
with the Bellman equations. Our theoretical guarantees only require Bellman
closedness as standard in the exploratory setting, in which case bonus-based
pessimism fails to provide guarantees. Even in the special case of linear
function approximation where stronger expressivity assumptions hold, our result
improves upon a recent bonus-based approach by $\mathcal{O}(d)$ in its sample
complexity when the action space is finite. Remarkably, our algorithms
automatically adapt to the best bias-variance tradeoff in the hindsight,
whereas most prior approaches require tuning extra hyperparameters a priori.
