CTDS: Centralized Teacher with Decentralized Student for Multi-Agent
  Reinforcement Learning
  Due to the partial observability and communication constraints in many
multi-agent reinforcement learning (MARL) tasks, centralized training with
decentralized execution (CTDE) has become one of the most widely used MARL
paradigms. In CTDE, centralized information is dedicated to learning the
allocation of the team reward with a mixing network, while the learning of
individual Q-values is usually based on local observations. The insufficient
utility of global observation will degrade performance in challenging
environments. To this end, this work proposes a novel Centralized Teacher with
Decentralized Student (CTDS) framework, which consists of a teacher model and a
student model. Specifically, the teacher model allocates the team reward by
learning individual Q-values conditioned on global observation, while the
student model utilizes the partial observations to approximate the Q-values
estimated by the teacher model. In this way, CTDS balances the full utilization
of global observation during training and the feasibility of decentralized
execution for online inference. Our CTDS framework is generic which is ready to
be applied upon existing CTDE methods to boost their performance. We conduct
experiments on a challenging set of StarCraft II micromanagement tasks to test
the effectiveness of our method and the results show that CTDS outperforms the
existing value-based MARL methods.
