Gradient play in stochastic games: stationary points, convergence, and
  sample complexity
  We study the performance of the gradient play algorithm for stochastic games
(SGs), where each agent tries to maximize its own total discounted reward by
making decisions independently based on current state information which is
shared between agents. Policies are directly parameterized by the probability
of choosing a certain action at a given state. We show that Nash equilibria
(NEs) and first-order stationary policies are equivalent in this setting, and
give a local convergence rate around strict NEs. Further, for a subclass of SGs
called Markov potential games (which includes the cooperative setting with
identical rewards among agents as an important special case), we design a
sample-based reinforcement learning algorithm and give a non-asymptotic global
convergence rate analysis for both exact gradient play and our sample-based
learning algorithm. Our result shows that the number of iterations to reach an
$\epsilon$-NE scales linearly, instead of exponentially, with the number of
agents. Local geometry and local stability are also considered, where we prove
that strict NEs are local maxima of the total potential function and
fully-mixed NEs are saddle points.
