Nonparametric Regression with Shallow Overparameterized Neural Networks
  Trained by GD with Early Stopping
  We explore the ability of overparameterized shallow neural networks to learn
Lipschitz regression functions with and without label noise when trained by
Gradient Descent (GD). To avoid the problem that in the presence of noisy
labels, neural networks trained to nearly zero training error are inconsistent
on this class, we propose an early stopping rule that allows us to show optimal
rates. This provides an alternative to the result of Hu et al. (2021) who
studied the performance of $\ell 2$ -regularized GD for training shallow
networks in nonparametric regression which fully relied on the infinite-width
network (Neural Tangent Kernel (NTK)) approximation. Here we present a simpler
analysis which is based on a partitioning argument of the input space (as in
the case of 1-nearest-neighbor rule) coupled with the fact that trained neural
networks are smooth with respect to their inputs when trained by GD. In the
noise-free case the proof does not rely on any kernelization and can be
regarded as a finite-width result. In the case of label noise, by slightly
modifying the proof, the noise is controlled using a technique of Yao, Rosasco,
and Caponnetto (2007).
