The Rate of Convergence of Variation-Constrained Deep Neural Networks
  Multi-layer feedforward networks have been used to approximate a wide range
of nonlinear functions. An important and fundamental problem is to understand
the learnability of a network model through its statistical risk, or the
expected prediction error on future data. To the best of our knowledge, the
rate of convergence of neural networks shown by existing works is bounded by at
most the order of $n^{-1/4}$ for a sample size of $n$. In this paper, we show
that a class of variation-constrained neural networks, with arbitrary width,
can achieve near-parametric rate $n^{-1/2+\delta}$ for an arbitrarily small
positive constant $\delta$. It is equivalent to $n^{-1 +2\delta}$ under the
mean squared error. This rate is also observed by numerical experiments. The
result indicates that the neural function space needed for approximating smooth
functions may not be as large as what is often perceived. Our result also
provides insight to the phenomena that deep neural networks do not easily
suffer from overfitting when the number of neurons and learning parameters
rapidly grow with $n$ or even surpass $n$. We also discuss the rate of
convergence regarding other network parameters, including the input dimension,
network layer, and coefficient norm.
