The Limit Points of (Optimistic) Gradient Descent in Min-Max
  Optimization
  Motivated by applications in Optimization, Game Theory, and the training of
Generative Adversarial Networks, the convergence properties of first order
methods in min-max problems have received extensive study. It has been
recognized that they may cycle, and there is no good understanding of their
limit points when they do not. When they converge, do they converge to local
min-max solutions? We characterize the limit points of two basic first order
methods, namely Gradient Descent/Ascent (GDA) and Optimistic Gradient Descent
Ascent (OGDA). We show that both dynamics avoid unstable critical points for
almost all initializations. Moreover, for small step sizes and under mild
assumptions, the set of \{OGDA\}-stable critical points is a superset of
\{GDA\}-stable critical points, which is a superset of local min-max solutions
(strict in some cases). The connecting thread is that the behavior of these
dynamics can be studied from a dynamical systems perspective.
