Multi-Agent Collaboration via Reward Attribution Decomposition
  Recent advances in multi-agent reinforcement learning (MARL) have achieved
super-human performance in games like Quake 3 and Dota 2. Unfortunately, these
techniques require orders-of-magnitude more training rounds than humans and
don't generalize to new agent configurations even on the same game. In this
work, we propose Collaborative Q-learning (CollaQ) that achieves
state-of-the-art performance in the StarCraft multi-agent challenge and
supports ad hoc team play. We first formulate multi-agent collaboration as a
joint optimization on reward assignment and show that each agent has an
approximately optimal policy that decomposes into two parts: one part that only
relies on the agent's own state, and the other part that is related to states
of nearby agents. Following this novel finding, CollaQ decomposes the
Q-function of each agent into a self term and an interactive term, with a
Multi-Agent Reward Attribution (MARA) loss that regularizes the training.
CollaQ is evaluated on various StarCraft maps and shows that it outperforms
existing state-of-the-art techniques (i.e., QMIX, QTRAN, and VDN) by improving
the win rate by 40% with the same number of samples. In the more challenging ad
hoc team play setting (i.e., reweight/add/remove units without re-training or
finetuning), CollaQ outperforms previous SoTA by over 30%.
