F2A2: Flexible Fully-decentralized Approximate Actor-critic for
  Cooperative Multi-agent Reinforcement Learning
  Traditional centralized multi-agent reinforcement learning (MARL) algorithms
are sometimes unpractical in complicated applications, due to non-interactivity
between agents, curse of dimensionality and computation complexity. Hence,
several decentralized MARL algorithms are motivated. However, existing
decentralized methods only handle the fully cooperative setting where massive
information needs to be transmitted in training. The block coordinate gradient
descent scheme they used for successive independent actor and critic steps can
simplify the calculation, but it causes serious bias. In this paper, we propose
a flexible fully decentralized actor-critic MARL framework, which can combine
most of actor-critic methods, and handle large-scale general cooperative
multi-agent setting. A primal-dual hybrid gradient descent type algorithm
framework is designed to learn individual agents separately for
decentralization. From the perspective of each agent, policy improvement and
value evaluation are jointly optimized, which can stabilize multi-agent policy
learning. Furthermore, our framework can achieve scalability and stability for
large-scale environment and reduce information transmission, by the parameter
sharing mechanism and a novel modeling-other-agents methods based on
theory-of-mind and online supervised learning. Sufficient experiments in
cooperative Multi-agent Particle Environment and StarCraft II show that our
decentralized MARL instantiation algorithms perform competitively against
conventional centralized and decentralized methods.
