Independent Natural Policy Gradient Methods for Potential Games:
  Finite-time Global Convergence with Entropy Regularization
  A major challenge in multi-agent systems is that the system complexity grows
dramatically with the number of agents as well as the size of their action
spaces, which is typical in real world scenarios such as autonomous vehicles,
robotic teams, network routing, etc. It is hence in imminent need to design
decentralized or independent algorithms where the update of each agent is only
based on their local observations without the need of introducing complex
communication/coordination mechanisms.
  In this work, we study the finite-time convergence of independent
entropy-regularized natural policy gradient (NPG) methods for potential games,
where the difference in an agent's utility function due to unilateral deviation
matches exactly that of a common potential function. The proposed
entropy-regularized NPG method enables each agent to deploy symmetric,
decentralized, and multiplicative updates according to its own payoff. We show
that the proposed method converges to the quantal response equilibrium (QRE) --
the equilibrium to the entropy-regularized game -- at a sublinear rate, which
is independent of the size of the action space and grows at most sublinearly
with the number of agents. Appealingly, the convergence rate further becomes
independent with the number of agents for the important special case of
identical-interest games, leading to the first method that converges at a
dimension-free rate. Our approach can be used as a smoothing technique to find
an approximate Nash equilibrium (NE) of the unregularized problem without
assuming that stationary policies are isolated.
