Credit Assignment with Meta-Policy Gradient for Multi-Agent
  Reinforcement Learning
  Reward decomposition is a critical problem in centralized training with
decentralized execution~(CTDE) paradigm for multi-agent reinforcement learning.
To take full advantage of global information, which exploits the states from
all agents and the related environment for decomposing Q values into individual
credits, we propose a general meta-learning-based Mixing Network with Meta
Policy Gradient~(MNMPG) framework to distill the global hierarchy for delicate
reward decomposition. The excitation signal for learning global hierarchy is
deduced from the episode reward difference between before and after "exercise
updates" through the utility network. Our method is generally applicable to the
CTDE method using a monotonic mixing network. Experiments on the StarCraft II
micromanagement benchmark demonstrate that our method just with a simple
utility network is able to outperform the current state-of-the-art MARL
algorithms on 4 of 5 super hard scenarios. Better performance can be further
achieved when combined with a role-based utility network.
