On Gradient-Based Learning in Continuous Games
  We formulate a general framework for competitive gradient-based learning that
encompasses a wide breadth of multi-agent learning algorithms, and analyze the
limiting behavior of competitive gradient-based learning algorithms using
dynamical systems theory. For both general-sum and potential games, we
characterize a non-negligible subset of the local Nash equilibria that will be
avoided if each agent employs a gradient-based learning algorithm. We also shed
light on the issue of convergence to non-Nash strategies in general- and
zero-sum games, which may have no relevance to the underlying game, and arise
solely due to the choice of algorithm. The existence and frequency of such
strategies may explain some of the difficulties encountered when using gradient
descent in zero-sum games as, e.g., in the training of generative adversarial
networks. To reinforce the theoretical contributions, we provide empirical
results that highlight the frequency of linear quadratic dynamic games (a
benchmark for multi-agent reinforcement learning) that admit global Nash
equilibria that are almost surely avoided by policy gradient.
