Stability and Generalization of Learning Algorithms that Converge to
  Global Optima
  We establish novel generalization bounds for learning algorithms that
converge to global minima. We do so by deriving black-box stability results
that only depend on the convergence of a learning algorithm and the geometry
around the minimizers of the loss function. The results are shown for nonconvex
loss functions satisfying the Polyak-{\L}ojasiewicz (PL) and the quadratic
growth (QG) conditions. We further show that these conditions arise for some
neural networks with linear activations. We use our black-box results to
establish the stability of optimization algorithms such as stochastic gradient
descent (SGD), gradient descent (GD), randomized coordinate descent (RCD), and
the stochastic variance reduced gradient method (SVRG), in both the PL and the
strongly convex setting. Our results match or improve state-of-the-art
generalization bounds and can easily be extended to similar optimization
algorithms. Finally, we show that although our results imply comparable
stability for SGD and GD in the PL setting, there exist simple neural networks
with multiple local minima where SGD is stable but GD is not.
