Adversarially Trained Actor Critic for Offline Reinforcement Learning
  We propose Adversarially Trained Actor Critic (ATAC), a new model-free
algorithm for offline reinforcement learning under insufficient data coverage,
based on a two-player Stackelberg game framing of offline RL: A policy actor
competes against an adversarially trained value critic, who finds
data-consistent scenarios where the actor is inferior to the data-collection
behavior policy. We prove that, when the actor attains no regret in the
two-player game, running ATAC produces a policy that provably 1) outperforms
the behavior policy over a wide range of hyperparameters, and 2) competes with
the best policy covered by data with appropriately chosen hyperparameters.
Compared with existing works, notably our framework offers both theoretical
guarantees for general function approximation and a deep RL implementation
scalable to complex environments and large datasets. In the D4RL benchmark,
ATAC consistently outperforms state-of-the-art offline RL algorithms on a range
of continuous control tasks
