On Generalization of Adaptive Methods for Over-parameterized Linear
  Regression
  Over-parameterization and adaptive methods have played a crucial role in the
success of deep learning in the last decade. The widespread use of
over-parameterization has forced us to rethink generalization by bringing forth
new phenomena, such as implicit regularization of optimization algorithms and
double descent with training progression. A series of recent works have started
to shed light on these areas in the quest to understand -- why do neural
networks generalize well? The setting of over-parameterized linear regression
has provided key insights into understanding this mysterious behavior of neural
networks.
  In this paper, we aim to characterize the performance of adaptive methods in
the over-parameterized linear regression setting. First, we focus on two
sub-classes of adaptive methods depending on their generalization performance.
For the first class of adaptive methods, the parameter vector remains in the
span of the data and converges to the minimum norm solution like gradient
descent (GD). On the other hand, for the second class of adaptive methods, the
gradient rotation caused by the pre-conditioner matrix results in an in-span
component of the parameter vector that converges to the minimum norm solution
and the out-of-span component that saturates. Our experiments on
over-parameterized linear regression and deep neural networks support this
theory.
