Towards an Understanding of Benign Overfitting in Neural Networks
  Modern machine learning models often employ a huge number of parameters and
are typically optimized to have zero training loss; yet surprisingly, they
possess near-optimal prediction performance, contradicting classical learning
theory. We examine how these benign overfitting phenomena occur in a two-layer
neural network setting where sample covariates are corrupted with noise. We
address the high dimensional regime, where the data dimension $d$ grows with
the number $n$ of data points. Our analysis combines an upper bound on the bias
with matching upper and lower bounds on the variance of the interpolator (an
estimator that interpolates the data). These results indicate that the excess
learning risk of the interpolator decays under mild conditions. We further show
that it is possible for the two-layer ReLU network interpolator to achieve a
near minimax-optimal learning rate, which to our knowledge is the first
generalization result for such networks. Finally, our theory predicts that the
excess learning risk starts to increase once the number of parameters $s$ grows
beyond $O(n^2)$, matching recent empirical findings.
