Rethinking the Implementation Tricks and Monotonicity Constraint in
  Cooperative Multi-Agent Reinforcement Learning
  Many complex multi-agent systems such as robot swarms control and autonomous
vehicle coordination can be modeled as Multi-Agent Reinforcement Learning
(MARL) tasks. QMIX, a widely popular MARL algorithm, has been used as a
baseline for the benchmark environments, e.g., Starcraft Multi-Agent Challenge
(SMAC), Difficulty-Enhanced Predator-Prey (DEPP). Recent variants of QMIX
target relaxing the monotonicity constraint of QMIX, allowing for performance
improvement in SMAC. In this paper, we investigate the code-level optimizations
of these variants and the monotonicity constraint. (1) We find that such
improvements of the variants are significantly affected by various code-level
optimizations. (2) The experiment results show that QMIX with normalized
optimizations outperforms other works in SMAC; (3) beyond the common wisdom
from these works, the monotonicity constraint can improve sample efficiency in
SMAC and DEPP. We also discuss why monotonicity constraints work well in purely
cooperative tasks with a theoretical analysis. We open-source the code at
\url{https://github.com/hijkzzz/pymarl2}.
