Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
  One of the preeminent obstacles to scaling multi-agent reinforcement learning
to large numbers of agents is assigning credit to individual agents' actions.
In this paper, we address this credit assignment problem with an approach that
we call \textit{partial reward decoupling} (PRD), which attempts to decompose
large cooperative multi-agent RL problems into decoupled subproblems involving
subsets of agents, thereby simplifying credit assignment. We empirically
demonstrate that decomposing the RL problem using PRD in an actor-critic
algorithm results in lower variance policy gradient estimates, which improves
data efficiency, learning stability, and asymptotic performance across a wide
array of multi-agent RL tasks, compared to various other actor-critic
approaches. Additionally, we relate our approach to counterfactual multi-agent
policy gradient (COMA), a state-of-the-art MARL algorithm, and empirically show
that our approach outperforms COMA by making better use of information in
agents' reward streams, and by enabling recent advances in advantage estimation
to be used.
