Contrasting Centralized and Decentralized Critics in Multi-Agent
  Reinforcement Learning
  Centralized Training for Decentralized Execution, where agents are trained
offline using centralized information but execute in a decentralized manner
online, has gained popularity in the multi-agent reinforcement learning
community. In particular, actor-critic methods with a centralized critic and
decentralized actors are a common instance of this idea. However, the
implications of using a centralized critic in this context are not fully
discussed and understood even though it is the standard choice of many
algorithms. We therefore formally analyze centralized and decentralized critic
approaches, providing a deeper understanding of the implications of critic
choice. Because our theory makes unrealistic assumptions, we also empirically
compare the centralized and decentralized critic methods over a wide set of
environments to validate our theories and to provide practical advice. We show
that there exist misconceptions regarding centralized critics in the current
literature and show that the centralized critic design is not strictly
beneficial, but rather both centralized and decentralized critics have
different pros and cons that should be taken into account by algorithm
designers.
