A priori generalization error for two-layer ReLU neural network through
  minimum norm solution
  We focus on estimating \emph{a priori} generalization error of two-layer ReLU
neural networks (NNs) trained by mean squared error, which only depends on
initial parameters and the target function, through the following research
line. We first estimate \emph{a priori} generalization error of finite-width
two-layer ReLU NN with constraint of minimal norm solution, which is proved by
\cite{zhang2019type} to be an equivalent solution of a linearized (w.r.t.
parameter) finite-width two-layer NN. As the width goes to infinity, the
linearized NN converges to the NN in Neural Tangent Kernel (NTK) regime
\citep{jacot2018neural}. Thus, we can derive the \emph{a priori} generalization
error of two-layer ReLU NN in NTK regime. The distance between NN in a NTK
regime and a finite-width NN with gradient training is estimated by
\cite{arora2019exact}. Based on the results in \cite{arora2019exact}, our work
proves an \emph{a priori} generalization error bound of two-layer ReLU NNs.
This estimate uses the intrinsic implicit bias of the minimum norm solution
without requiring extra regularity in the loss function. This \emph{a priori}
estimate also implies that NN does not suffer from curse of dimensionality, and
a small generalization error can be achieved without requiring exponentially
large number of neurons. In addition the research line proposed in this paper
can also be used to study other properties of the finite-width network, such as
the posterior generalization error.
