On Improving Model-Free Algorithms for Decentralized Multi-Agent
  Reinforcement Learning
  Multi-agent reinforcement learning (MARL) algorithms often suffer from an
exponential sample complexity dependence on the number of agents, a phenomenon
known as \emph{the curse of multiagents}. In this paper, we address this
challenge by investigating sample-efficient model-free algorithms in
\emph{decentralized} MARL, and aim to improve existing algorithms along this
line. For learning (coarse) correlated equilibria in general-sum Markov games,
we propose \emph{stage-based} V-learning algorithms that significantly simplify
the algorithmic design and analysis of recent works, and circumvent a rather
complicated no-\emph{weighted}-regret bandit subroutine. For learning Nash
equilibria in Markov potential games, we propose an independent policy gradient
algorithm with a decentralized momentum-based variance reduction technique. All
our algorithms are decentralized in that each agent can make decisions based on
only its local information. Neither communication nor centralized coordination
is required during learning, leading to a natural generalization to a large
number of agents. We also provide numerical simulations to corroborate our
theoretical findings.
