Near-Minimax Optimal Estimation With Shallow ReLU Neural Networks
  We study the problem of estimating an unknown function from noisy data using
shallow ReLU neural networks. The estimators we study minimize the sum of
squared data-fitting errors plus a regularization term proportional to the
squared Euclidean norm of the network weights. This minimization corresponds to
the common approach of training a neural network with weight decay. We quantify
the performance (mean-squared error) of these neural network estimators when
the data-generating function belongs to the second-order Radon-domain bounded
variation space. This space of functions was recently proposed as the natural
function space associated with shallow ReLU neural networks. We derive a
minimax lower bound for the estimation problem for this function space and show
that the neural network estimators are minimax optimal up to logarithmic
factors. This minimax rate is immune to the curse of dimensionality. We
quantify an explicit gap between neural networks and linear methods (which
include kernel methods) by deriving a linear minimax lower bound for the
estimation problem, showing that linear methods necessarily suffer the curse of
dimensionality in this function space. As a result, this paper sheds light on
the phenomenon that neural networks seem to break the curse of dimensionality.
