Relaxing the Feature Covariance Assumption: Time-Variant Bounds for
  Benign Overfitting in Linear Regression
  Benign overfitting demonstrates that overparameterized models can perform
well on test data while fitting noisy training data. However, it only considers
the final min-norm solution in linear regression, which ignores the algorithm
information and the corresponding training procedure. In this paper, we
generalize the idea of benign overfitting to the whole training trajectory
instead of the min-norm solution and derive a time-variant bound based on the
trajectory analysis. Starting from the time-variant bound, we further derive a
time interval that suffices to guarantee a consistent generalization error for
a given feature covariance. Unlike existing approaches, the newly proposed
generalization bound is characterized by a time-variant effective dimension of
feature covariance. By introducing the time factor, we relax the strict
assumption on the feature covariance matrix required in previous benign
overfitting under the regimes of overparameterized linear regression with
gradient descent. This paper extends the scope of benign overfitting, and
experiment results indicate that the proposed bound accords better with
empirical evidence.
