Sharp Lower Bounds on the Approximation Rate of Shallow Neural Networks
  We consider the approximation rates of shallow neural networks with respect
to the variation norm. Upper bounds on these rates have been established for
sigmoidal and ReLU activation functions, but it has remained an important open
problem whether these rates are sharp. In this article, we provide a solution
to this problem by proving sharp lower bounds on the approximation rates for
shallow neural networks, which are obtained by lower bounding the $L^2$-metric
entropy of the convex hull of the neural network basis functions. In addition,
our methods also give sharp lower bounds on the Kolmogorov $n$-widths of this
convex hull, which show that the variation spaces corresponding to shallow
neural networks cannot be efficiently approximated by linear methods. These
lower bounds apply to both sigmoidal activation functions with bounded
variation and to activation functions which are a power of the ReLU. Our
results also quantify how much stronger the Barron spectral norm is than the
variation norm and, combined with previous results, give the asymptotics of the
$L^\infty$-metric entropy up to logarithmic factors in the case of the ReLU
activation function.
