Learning Implicit Credit Assignment for Cooperative Multi-Agent
  Reinforcement Learning
  We present a multi-agent actor-critic method that aims to implicitly address
the credit assignment problem under fully cooperative settings. Our key
motivation is that credit assignment among agents may not require an explicit
formulation as long as (1) the policy gradients derived from a centralized
critic carry sufficient information for the decentralized agents to maximize
their joint action value through optimal cooperation and (2) a sustained level
of exploration is enforced throughout training. Under the centralized training
with decentralized execution (CTDE) paradigm, we achieve the former by
formulating the centralized critic as a hypernetwork such that a latent state
representation is integrated into the policy gradients through its
multiplicative association with the stochastic policies; to achieve the latter,
we derive a simple technique called adaptive entropy regularization where
magnitudes of the entropy gradients are dynamically rescaled based on the
current policy stochasticity to encourage consistent levels of exploration. Our
algorithm, referred to as LICA, is evaluated on several benchmarks including
the multi-agent particle environments and a set of challenging StarCraft II
micromanagement tasks, and we show that LICA significantly outperforms previous
methods.
