The Landscape of the Proximal Point Method for Nonconvex-Nonconcave
  Minimax Optimization
  Minimax optimization has become a central tool in machine learning with
applications in robust optimization, reinforcement learning, GANs, etc. These
applications are often nonconvex-nonconcave, but the existing theory is unable
to identify and deal with the fundamental difficulties this poses. In this
paper, we study the classic proximal point method (PPM) applied to
nonconvex-nonconcave minimax problems. We find that a classic generalization of
the Moreau envelope by Attouch and Wets provides key insights. Critically, we
show this envelope not only smooths the objective but can convexify and
concavify it based on the level of interaction present between the minimizing
and maximizing variables. From this, we identify three distinct regions of
nonconvex-nonconcave problems. When interaction is sufficiently strong, we
derive global linear convergence guarantees. Conversely when the interaction is
fairly weak, we derive local linear convergence guarantees with a proper
initialization. Between these two settings, we show that PPM may diverge or
converge to a limit cycle.
