Learning to Coordinate in Multi-Agent Systems: A Coordinated
  Actor-Critic Algorithm and Finite-Time Guarantees
  Multi-agent reinforcement learning (MARL) has attracted much research
attention recently. However, unlike its single-agent counterpart, many
theoretical and algorithmic aspects of MARL have not been well-understood. In
this paper, we study the emergence of coordinated behavior by autonomous agents
using an actor-critic (AC) algorithm. Specifically, we propose and analyze a
class of coordinated actor-critic algorithms (CAC) in which individually
parametrized policies have a {\it shared} part (which is jointly optimized
among all agents) and a {\it personalized} part (which is only locally
optimized). Such kind of {\it partially personalized} policy allows agents to
learn to coordinate by leveraging peers' past experience and adapt to
individual tasks. The flexibility in our design allows the proposed MARL-CAC
algorithm to be used in a {\it fully decentralized} setting, where the agents
can only communicate with their neighbors, as well as a {\it federated}
setting, where the agents occasionally communicate with a server while
optimizing their (partially personalized) local models. Theoretically, we show
that under some standard regularity assumptions, the proposed MARL-CAC
algorithm requires $\mathcal{O}(\epsilon^{-\frac{5}{2}})$ samples to achieve an
$\epsilon$-stationary solution (defined as the solution whose squared norm of
the gradient of the objective function is less than $\epsilon$). To the best of
our knowledge, this work provides the first finite-sample guarantee for
decentralized AC algorithm with partially personalized policies.
