A Game-Theoretic Approach to Multi-Agent Trust Region Optimization
  Trust region methods are widely applied in single-agent reinforcement
learning problems due to their monotonic performance-improvement guarantee at
every iteration. Nonetheless, when applied in multi-agent settings, the
guarantee of trust region methods no longer holds because an agent's payoff is
also affected by other agents' adaptive behaviors. To tackle this problem, we
conduct a game-theoretical analysis in the policy space, and propose a
multi-agent trust region learning method (MATRL), which enables trust region
optimization for multi-agent learning. Specifically, MATRL finds a stable
improvement direction that is guided by the solution concept of Nash
equilibrium at the meta-game level. We derive the monotonic improvement
guarantee in multi-agent settings and empirically show the local convergence of
MATRL to stable fixed points in the two-player rotational differential game. To
test our method, we evaluate MATRL in both discrete and continuous multiplayer
general-sum games including checker and switch grid worlds, multi-agent MuJoCo,
and Atari games. Results suggest that MATRL significantly outperforms strong
multi-agent reinforcement learning baselines.
