Learning Meta Representations for Agents in Multi-Agent Reinforcement
  Learning
  In multi-agent reinforcement learning, the behaviors that agents learn in a
single Markov Game (MG) are typically confined to the given agent number (i.e.,
population size). Every single MG induced by varying population sizes may
possess distinct optimal joint strategies and game-specific knowledge, which
are modeled independently in modern multi-agent algorithms. In this work, we
focus on creating agents that generalize across population-varying MGs. Instead
of learning a unimodal policy, each agent learns a policy set that is formed by
effective strategies across a variety of games. We propose Meta Representations
for Agents (MRA) that explicitly models the game-common and game-specific
strategic knowledge. By representing the policy sets with multi-modal latent
policies, the common strategic knowledge and diverse strategic modes are
discovered with an iterative optimization procedure. We prove that as an
approximation to a constrained mutual information maximization objective, the
learned policies can reach Nash Equilibrium in every evaluation MG under the
assumption of Lipschitz game on a sufficiently large latent space. When
deploying it at practical latent models with limited size, fast adaptation can
be achieved by leveraging the first-order gradient information. Extensive
experiments show the effectiveness of MRA on both training performance and
generalization ability in hard and unseen games.
