The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
  Modern reinforcement learning (RL) commonly engages practical problems with
large state spaces, where function approximation must be deployed to
approximate either the value function or the policy. While recent progresses in
RL theory address a rich set of RL problems with general function
approximation, such successes are mostly restricted to the single-agent
setting. It remains elusive how to extend these results to multi-agent RL,
especially due to the new challenges arising from its game-theoretical nature.
This paper considers two-player zero-sum Markov Games (MGs). We propose a new
algorithm that can provably find the Nash equilibrium policy using a polynomial
number of samples, for any MG with low multi-agent Bellman-Eluder dimension --
a new complexity measure adapted from its single-agent version (Jin et al.,
2021). A key component of our new algorithm is the exploiter, which facilitates
the learning of the main player by deliberately exploiting her weakness. Our
theoretical framework is generic, which applies to a wide range of models
including but not limited to tabular MGs, MGs with linear or kernel function
approximation, and MGs with rich observations.
