Beyond Lipschitz: Sharp Generalization and Excess Risk Bounds for
  Full-Batch GD
  We provide sharp path-dependent generalization and excess risk guarantees for
the full-batch Gradient Descent (GD) algorithm on smooth losses (possibly
non-Lipschitz, possibly nonconvex), under an interpolation regime. At the heart
of our analysis is a new generalization error bound for deterministic symmetric
algorithms, which implies that average output stability and a bounded expected
optimization error at termination lead to generalization. This result shows
that small generalization error occurs along the optimization path, and allows
us to bypass Lipschitz or sub-Gaussian assumptions on the loss prevalent in
previous works. For nonconvex, Polyak-Lojasiewicz (PL), convex and strongly
convex losses, we show the explicit dependence of the generalization error in
terms of the accumulated path-dependent optimization error, terminal
optimization error, number of samples, and number of iterations. For nonconvex
smooth losses, we prove that full-batch GD efficiently generalizes close to any
stationary point at termination, under the proper choice of a decreasing step
size. Further, if the loss is nonconvex but the objective is PL, we derive
quadratically vanishing bounds on the generalization error and the
corresponding excess risk, for a choice of a large constant step size. For
(resp. strongly-) convex smooth losses, we prove that full-batch GD also
generalizes for large constant step sizes, and achieves (resp. quadratically)
small excess risk while training fast. In all cases, we close the
generalization error gap, by showing matching generalization and optimization
error rates. Our full-batch GD generalization error and excess risk bounds are
strictly tighter than existing bounds for (stochastic) GD, when the loss is
smooth (but possibly non-Lipschitz).
