Deep Implicit Coordination Graphs for Multi-agent Reinforcement Learning
  Multi-agent reinforcement learning (MARL) requires coordination to
efficiently solve certain tasks. Fully centralized control is often infeasible
in such domains due to the size of joint action spaces. Coordination graph
based formalization allows reasoning about the joint action based on the
structure of interactions. However, they often require domain expertise in
their design. This paper introduces the deep implicit coordination graph (DICG)
architecture for such scenarios. DICG consists of a module for inferring the
dynamic coordination graph structure which is then used by a graph neural
network based module to learn to implicitly reason about the joint actions or
values. DICG allows learning the tradeoff between full centralization and
decentralization via standard actor-critic methods to significantly improve
coordination for domains with large number of agents. We apply DICG to both
centralized-training-centralized-execution and
centralized-training-decentralized-execution regimes. We demonstrate that DICG
solves the relative overgeneralization pathology in predatory-prey tasks as
well as outperforms various MARL baselines on the challenging StarCraft II
Multi-agent Challenge (SMAC) and traffic junction environments.
