Provable Benefits of Actor-Critic Methods for Offline Reinforcement
  Learning
  Actor-critic methods are widely used in offline reinforcement learning
practice, but are not so well-understood theoretically. We propose a new
offline actor-critic algorithm that naturally incorporates the pessimism
principle, leading to several key advantages compared to the state of the art.
The algorithm can operate when the Bellman evaluation operator is closed with
respect to the action value function of the actor's policies; this is a more
general setting than the low-rank MDP model. Despite the added generality, the
procedure is computationally tractable as it involves the solution of a
sequence of second-order programs. We prove an upper bound on the suboptimality
gap of the policy returned by the procedure that depends on the data coverage
of any arbitrary, possibly data dependent comparator policy. The achievable
guarantee is complemented with a minimax lower bound that is matching up to
logarithmic factors.
