Finite-Sample Analysis of Decentralized Q-Learning for Stochastic Games
  Learning in stochastic games is arguably the most standard and fundamental
setting in multi-agent reinforcement learning (MARL). In this paper, we
consider decentralized MARL in stochastic games in the non-asymptotic regime.
In particular, we establish the finite-sample complexity of fully decentralized
Q-learning algorithms in a significant class of general-sum stochastic games
(SGs) - weakly acyclic SGs, which includes the common cooperative MARL setting
with an identical reward to all agents (a Markov team problem) as a special
case. We focus on the practical while challenging setting of fully
decentralized MARL, where neither the rewards nor the actions of other agents
can be observed by each agent. In fact, each agent is completely oblivious to
the presence of other decision makers. Both the tabular and the linear function
approximation cases have been considered. In the tabular setting, we analyze
the sample complexity for the decentralized Q-learning algorithm to converge to
a Markov perfect equilibrium (Nash equilibrium). With linear function
approximation, the results are for convergence to a linear approximated
equilibrium - a new notion of equilibrium that we propose - which describes
that each agent's policy is a best reply (to other agents) within a linear
space. Numerical experiments are also provided for both settings to demonstrate
the results.
