Towards Understanding Cooperative Multi-Agent Q-Learning with Value
  Factorization
  Value factorization is a popular and promising approach to scaling up
multi-agent reinforcement learning in cooperative settings, which balances the
learning scalability and the representational capacity of value functions.
However, the theoretical understanding of such methods is limited. In this
paper, we formalize a multi-agent fitted Q-iteration framework for analyzing
factorized multi-agent Q-learning. Based on this framework, we investigate
linear value factorization and reveal that multi-agent Q-learning with this
simple decomposition implicitly realizes a powerful counterfactual credit
assignment, but may not converge in some settings. Through further analysis, we
find that on-policy training or richer joint value function classes can improve
its local or global convergence properties, respectively. Finally, to support
our theoretical implications in practical realization, we conduct an empirical
analysis of state-of-the-art deep multi-agent Q-learning algorithms on didactic
examples and a broad set of StarCraft II unit micromanagement tasks.
