Divergence-Regularized Multi-Agent Actor-Critic
  Entropy regularization is a popular method in reinforcement learning (RL).
Although it has many advantages, it alters the RL objective and makes the
converged policy deviate from the optimal policy of the original Markov
Decision Process. Though divergence regularization has been proposed to settle
this problem, it cannot be trivially applied to cooperative multi-agent
reinforcement learning (MARL). In this paper, we investigate divergence
regularization in cooperative MARL and propose a novel off-policy cooperative
MARL framework, divergence-regularized multi-agent actor-critic (DMAC).
Mathematically, we derive the update rule of DMAC which is naturally
off-policy, guarantees a monotonic policy improvement and is not biased by the
regularization. DMAC is a flexible framework and can be combined with many
existing MARL algorithms. We evaluate DMAC in a didactic stochastic game and
StarCraft Multi-Agent Challenge and empirically show that DMAC substantially
improves the performance of existing MARL algorithms.
