Implicit Bias of MSE Gradient Optimization in Underparameterized Neural
  Networks
  We study the dynamics of a neural network in function space when optimizing
the mean squared error via gradient flow. We show that in the
underparameterized regime the network learns eigenfunctions of an integral
operator $T_{K^\infty}$ determined by the Neural Tangent Kernel (NTK) at rates
corresponding to their eigenvalues. For example, for uniformly distributed data
on the sphere $S^{d - 1}$ and rotation invariant weight distributions, the
eigenfunctions of $T_{K^\infty}$ are the spherical harmonics. Our results can
be understood as describing a spectral bias in the underparameterized regime.
The proofs use the concept of "Damped Deviations", where deviations of the NTK
matter less for eigendirections with large eigenvalues due to the occurence of
a damping factor. Aside from the underparameterized regime, the damped
deviations point-of-view can be used to track the dynamics of the empirical
risk in the overparameterized setting, allowing us to extend certain results in
the literature. We conclude that damped deviations offers a simple and unifying
perspective of the dynamics when optimizing the squared error.
