Multi-Agent MDP Homomorphic Networks
  This paper introduces Multi-Agent MDP Homomorphic Networks, a class of
networks that allows distributed execution using only local information, yet is
able to share experience between global symmetries in the joint state-action
space of cooperative multi-agent systems. In cooperative multi-agent systems,
complex symmetries arise between different configurations of the agents and
their local observations. For example, consider a group of agents navigating:
rotating the state globally results in a permutation of the optimal joint
policy. Existing work on symmetries in single agent reinforcement learning can
only be generalized to the fully centralized setting, because such approaches
rely on the global symmetry in the full state-action spaces, and these can
result in correspondences across agents. To encode such symmetries while still
allowing distributed execution we propose a factorization that decomposes
global symmetries into local transformations. Our proposed factorization allows
for distributing the computation that enforces global symmetries over local
agents and local interactions. We introduce a multi-agent equivariant policy
network based on this factorization. We show empirically on symmetric
multi-agent problems that globally symmetric distributable policies improve
data efficiency compared to non-equivariant baselines.
