Offline Reinforcement Learning with Realizability and Single-policy
  Concentrability
  Sample-efficiency guarantees for offline reinforcement learning (RL) often
rely on strong assumptions on both the function classes (e.g.,
Bellman-completeness) and the data coverage (e.g., all-policy concentrability).
Despite the recent efforts on relaxing these assumptions, existing works are
only able to relax one of the two factors, leaving the strong assumption on the
other factor intact. As an important open problem, can we achieve
sample-efficient offline RL with weak assumptions on both factors?
  In this paper we answer the question in the positive. We analyze a simple
algorithm based on the primal-dual formulation of MDPs, where the dual
variables (discounted occupancy) are modeled using a density-ratio function
against offline data. With proper regularization, we show that the algorithm
enjoys polynomial sample complexity, under only realizability and single-policy
concentrability. We also provide alternative analyses based on different
assumptions to shed light on the nature of primal-dual algorithms for offline
RL.
