A Comparative Analysis of the Optimization and Generalization Property
  of Two-layer Neural Network and Random Feature Models Under Gradient Descent
  Dynamics
  A fairly comprehensive analysis is presented for the gradient descent
dynamics for training two-layer neural network models in the situation when the
parameters in both layers are updated. General initialization schemes as well
as general regimes for the network width and training data size are considered.
In the over-parametrized regime, it is shown that gradient descent dynamics can
achieve zero training loss exponentially fast regardless of the quality of the
labels. In addition, it is proved that throughout the training process the
functions represented by the neural network model are uniformly close to that
of a kernel method. For general values of the network width and training data
size, sharp estimates of the generalization error is established for target
functions in the appropriate reproducing kernel Hilbert space.
