Benign Overfitting without Linearity: Neural Network Classifiers Trained
  by Gradient Descent for Noisy Linear Data
  Benign overfitting, the phenomenon where interpolating models generalize well
in the presence of noisy data, was first observed in neural network models
trained with gradient descent. To better understand this empirical observation,
we consider the generalization error of two-layer neural networks trained to
interpolation by gradient descent on the logistic loss following random
initialization. We assume the data comes from well-separated class-conditional
log-concave distributions and allow for a constant fraction of the training
labels to be corrupted by an adversary. We show that in this setting, neural
networks exhibit benign overfitting: they can be driven to zero training error,
perfectly fitting any noisy training labels, and simultaneously achieve test
error close to the Bayes-optimal error. In contrast to previous work on benign
overfitting that require linear or kernel-based predictors, our analysis holds
in a setting where both the model and learning dynamics are fundamentally
nonlinear.
