FACMAC: Factored Multi-Agent Centralised Policy Gradients
  We propose FACtored Multi-Agent Centralised policy gradients (FACMAC), a new
method for cooperative multi-agent reinforcement learning in both discrete and
continuous action spaces. Like MADDPG, a popular multi-agent actor-critic
method, our approach uses deep deterministic policy gradients to learn
policies. However, FACMAC learns a centralised but factored critic, which
combines per-agent utilities into the joint action-value function via a
non-linear monotonic function, as in QMIX, a popular multi-agent Q-learning
algorithm. However, unlike QMIX, there are no inherent constraints on factoring
the critic. We thus also employ a nonmonotonic factorisation and empirically
demonstrate that its increased representational capacity allows it to solve
some tasks that cannot be solved with monolithic, or monotonically factored
critics. In addition, FACMAC uses a centralised policy gradient estimator that
optimises over the entire joint action space, rather than optimising over each
agent's action space separately as in MADDPG. This allows for more coordinated
policy changes and fully reaps the benefits of a centralised critic. We
evaluate FACMAC on variants of the multi-agent particle environments, a novel
multi-agent MuJoCo benchmark, and a challenging set of StarCraft II
micromanagement tasks. Empirical results demonstrate FACMAC's superior
performance over MADDPG and other baselines on all three domains.
