QTRAN: Learning to Factorize with Transformation for Cooperative
  Multi-Agent Reinforcement Learning
  We explore value-based solutions for multi-agent reinforcement learning
(MARL) tasks in the centralized training with decentralized execution (CTDE)
regime popularized recently. However, VDN and QMIX are representative examples
that use the idea of factorization of the joint action-value function into
individual ones for decentralized execution. VDN and QMIX address only a
fraction of factorizable MARL tasks due to their structural constraint in
factorization such as additivity and monotonicity. In this paper, we propose a
new factorization method for MARL, QTRAN, which is free from such structural
constraints and takes on a new approach to transforming the original joint
action-value function into an easily factorizable one, with the same optimal
actions. QTRAN guarantees more general factorization than VDN or QMIX, thus
covering a much wider class of MARL tasks than does previous methods. Our
experiments for the tasks of multi-domain Gaussian-squeeze and modified
predator-prey demonstrate QTRAN's superior performance with especially larger
margins in games whose payoffs penalize non-cooperative behavior more
aggressively.
