Efficient Competitive Self-Play Policy Optimization
  Reinforcement learning from self-play has recently reported many successes.
Self-play, where the agents compete with themselves, is often used to generate
training data for iterative policy improvement. In previous work, heuristic
rules are designed to choose an opponent for the current learner. Typical rules
include choosing the latest agent, the best agent, or a random historical
agent. However, these rules may be inefficient in practice and sometimes do not
guarantee convergence even in the simplest matrix games. In this paper, we
propose a new algorithmic framework for competitive self-play reinforcement
learning in two-player zero-sum games. We recognize the fact that the Nash
equilibrium coincides with the saddle point of the stochastic payoff function,
which motivates us to borrow ideas from classical saddle point optimization
literature. Our method trains several agents simultaneously, and intelligently
takes each other as opponent based on simple adversarial rules derived from a
principled perturbation-based saddle optimization method. We prove
theoretically that our algorithm converges to an approximate equilibrium with
high probability in convex-concave games under standard assumptions. Beyond the
theory, we further show the empirical superiority of our method over baseline
methods relying on the aforementioned opponent-selection heuristics in matrix
games, grid-world soccer, Gomoku, and simulated robot sumo, with neural net
policy function approximators.
