Finite-Sample Analysis For Decentralized Batch Multi-Agent Reinforcement
  Learning With Networked Agents
  Despite the increasing interest in multi-agent reinforcement learning (MARL)
in multiple communities, understanding its theoretical foundation has long been
recognized as a challenging problem. In this work, we address this problem by
providing a finite-sample analysis for decentralized batch MARL with networked
agents. Specifically, we consider two decentralized MARL settings, where teams
of agents are connected by time-varying communication networks, and either
collaborate or compete in a zero-sum game setting, without any central
controller. These settings cover many conventional MARL settings in the
literature. For both settings, we develop batch MARL algorithms that can be
implemented in a decentralized fashion, and quantify the finite-sample errors
of the estimated action-value functions. Our error analysis captures how the
function class, the number of samples within each iteration, and the number of
iterations determine the statistical accuracy of the proposed algorithms. Our
results, compared to the finite-sample bounds for single-agent RL, involve
additional error terms caused by decentralized computation, which is inherent
in our decentralized MARL setting. This work appears to be the first
finite-sample analysis for batch MARL, a step towards rigorous theoretical
understanding of general MARL algorithms in the finite-sample regime.
