Monotonic Improvement Guarantees under Non-stationarity for
  Decentralized PPO
  We present a new monotonic improvement guarantee for optimizing decentralized
policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds
even when the transition dynamics are non-stationary. This new analysis
provides a theoretical understanding of the strong performance of two recent
actor-critic methods for MARL, i.e., Independent Proximal Policy Optimization
(IPPO) and Multi-Agent PPO (MAPPO), which both rely on independent ratios,
i.e., computing probability ratios separately for each agent's policy. We show
that, despite the non-stationarity that independent ratios cause, a monotonic
improvement guarantee still arises as a result of enforcing the trust region
constraint over all decentralized policies. We also show this trust region
constraint can be effectively enforced in a principled way by bounding
independent ratios based on the number of agents in training, providing a
theoretical foundation for proximal ratio clipping. Moreover, we show that the
surrogate objectives optimized in IPPO and MAPPO are essentially equivalent
when their critics converge to a fixed point. Finally, our empirical results
support the hypothesis that the strong performance of IPPO and MAPPO is a
direct result of enforcing such a trust region constraint via clipping in
centralized training, and the good values of the hyperparameters for this
enforcement are highly sensitive to the number of agents, as predicted by our
theoretical analysis.
