Stable Opponent Shaping in Differentiable Games
  A growing number of learning methods are actually differentiable games whose
players optimise multiple, interdependent objectives in parallel -- from GANs
and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful
approach to improve learning dynamics in these games, accounting for player
influence on others' updates. Learning with Opponent-Learning Awareness (LOLA)
is a recent algorithm that exploits this response and leads to cooperation in
settings like the Iterated Prisoner's Dilemma. Although experimentally
successful, we show that LOLA agents can exhibit 'arrogant' behaviour directly
at odds with convergence. In fact, remarkably few algorithms have theoretical
guarantees applying across all (n-player, non-convex) games. In this paper we
present Stable Opponent Shaping (SOS), a new method that interpolates between
LOLA and a stable variant named LookAhead. We prove that LookAhead converges
locally to equilibria and avoids strict saddles in all differentiable games.
SOS inherits these essential guarantees, while also shaping the learning of
opponents and consistently either matching or outperforming LOLA
experimentally.
