On the Complexity of Computing Markov Perfect Equilibrium in General-Sum
  Stochastic Games
  Similar to the role of Markov decision processes in reinforcement learning,
Stochastic Games (SGs) lay the foundation for the study of multi-agent
reinforcement learning (MARL) and sequential agent interactions. In this paper,
we derive that computing an approximate Markov Perfect Equilibrium (MPE) in a
finite-state discounted Stochastic Game within the exponential precision is
\textbf{PPAD}-complete. We adopt a function with a polynomially bounded
description in the strategy space to convert the MPE computation to a
fixed-point problem, even though the stochastic game may demand an exponential
number of pure strategies, in the number of states, for each agent. The
completeness result follows the reduction of the fixed-point problem to {\sc
End of the Line}. Our results indicate that finding an MPE in SGs is highly
unlikely to be \textbf{NP}-hard unless \textbf{NP}=\textbf{co-NP}. Our work
offers confidence for MARL research to study MPE computation on general-sum SGs
and to develop fruitful algorithms as currently on zero-sum SGs.
