On the Benefits of Large Learning Rates for Kernel Methods
  This paper studies an intriguing phenomenon related to the good
generalization performance of estimators obtained by using large learning rates
within gradient descent algorithms. First observed in the deep learning
literature, we show that a phenomenon can be precisely characterized in the
context of kernel methods, even though the resulting optimization problem is
convex. Specifically, we consider the minimization of a quadratic objective in
a separable Hilbert space, and show that with early stopping, the choice of
learning rate influences the spectral decomposition of the obtained solution on
the Hessian's eigenvectors. This extends an intuition described by Nakkiran
(2020) on a two-dimensional toy problem to realistic learning scenarios such as
kernel ridge regression. While large learning rates may be proven beneficial as
soon as there is a mismatch between the train and test objectives, we further
explain why it already occurs in classification tasks without assuming any
particular mismatch between train and test data distributions.
