MCMARL: Parameterizing Value Function via Mixture of Categorical
  Distributions for Multi-Agent Reinforcement Learning
  In cooperative multi-agent tasks, a team of agents jointly interact with an
environment by taking actions, receiving a team reward and observing the next
state. During the interactions, the uncertainty of environment and reward will
inevitably induce stochasticity in the long-term returns and the randomness can
be exacerbated with the increasing number of agents. However, such randomness
is ignored by most of the existing value-based multi-agent reinforcement
learning (MARL) methods, which only model the expectation of Q-value for both
individual agents and the team. Compared to using the expectations of the
long-term returns, it is preferable to directly model the stochasticity by
estimating the returns through distributions. With this motivation, this work
proposes a novel value-based MARL framework from a distributional perspective,
\emph{i.e.}, parameterizing value function via \underline{M}ixture of
\underline{C}ategorical distributions for MARL. Specifically, we model both
individual Q-values and global Q-value with categorical distribution. To
integrate categorical distributions, we define five basic operations on the
distribution, which allow the generalization of expected value function
factorization methods (\emph{e.g.}, VDN and QMIX) to their MCMARL variants. We
further prove that our MCMARL framework satisfies
\emph{Distributional-Individual-Global-Max} (DIGM) principle with respect to
the expectation of distribution, which guarantees the consistency between joint
and individual greedy action selections in the global Q-value and individual
Q-values. Empirically, we evaluate MCMARL on both a stochastic matrix game and
a challenging set of StarCraft II micromanagement tasks, showing the efficacy
of our framework.
