Efficient Model-based Multi-agent Reinforcement Learning via Optimistic
  Equilibrium Computation
  We consider model-based multi-agent reinforcement learning, where the
environment transition model is unknown and can only be learned via expensive
interactions with the environment. We propose H-MARL (Hallucinated Multi-Agent
Reinforcement Learning), a novel sample-efficient algorithm that can
efficiently balance exploration, i.e., learning about the environment, and
exploitation, i.e., achieve good equilibrium performance in the underlying
general-sum Markov game. H-MARL builds high-probability confidence intervals
around the unknown transition model and sequentially updates them based on
newly observed data. Using these, it constructs an optimistic hallucinated game
for the agents for which equilibrium policies are computed at each round. We
consider general statistical models (e.g., Gaussian processes, deep ensembles,
etc.) and policy classes (e.g., deep neural networks), and theoretically
analyze our approach by bounding the agents' dynamic regret. Moreover, we
provide a convergence rate to the equilibria of the underlying Markov game. We
demonstrate our approach experimentally on an autonomous driving simulation
benchmark. H-MARL learns successful equilibrium policies after a few
interactions with the environment and can significantly improve the performance
compared to non-exploratory methods.
