The Complexity of Markov Equilibrium in Stochastic Games
  We show that computing approximate stationary Markov coarse correlated
equilibria (CCE) in general-sum stochastic games is computationally
intractable, even when there are two players, the game is turn-based, the
discount factor is an absolute constant, and the approximation is an absolute
constant. Our intractability results stand in sharp contrast to normal-form
games where exact CCEs are efficiently computable. A fortiori, our results
imply that there are no efficient algorithms for learning stationary Markov CCE
policies in multi-agent reinforcement learning (MARL), even when the
interaction is two-player and turn-based, and both the discount factor and the
desired approximation of the learned policies is an absolute constant. In turn,
these results stand in sharp contrast to single-agent reinforcement learning
(RL) where near-optimal stationary Markov policies can be efficiently learned.
Complementing our intractability results for stationary Markov CCEs, we provide
a decentralized algorithm (assuming shared randomness among players) for
learning a nonstationary Markov CCE policy with polynomial time and sample
complexity in all problem parameters. Previous work for learning Markov CCE
policies all required exponential time and sample complexity in the number of
players.
