Decentralized Deterministic Multi-Agent Reinforcement Learning
  [Zhang, ICML 2018] provided the first decentralized actor-critic algorithm
for multi-agent reinforcement learning (MARL) that offers convergence
guarantees. In that work, policies are stochastic and are defined on finite
action spaces. We extend those results to offer a provably-convergent
decentralized actor-critic algorithm for learning deterministic policies on
continuous action spaces. Deterministic policies are important in real-world
settings. To handle the lack of exploration inherent in deterministic policies,
we consider both off-policy and on-policy settings. We provide the expression
of a local deterministic policy gradient, decentralized deterministic
actor-critic algorithms and convergence guarantees for linearly-approximated
value functions. This work will help enable decentralized MARL in
high-dimensional action spaces and pave the way for more widespread use of
MARL.
