Minimax Lower Bounds for Ridge Combinations Including Neural Nets
  Estimation of functions of $ d $ variables is considered using ridge
combinations of the form $ \textstyle\sum_{k=1}^m c_{1,k}
\phi(\textstyle\sum_{j=1}^d c_{0,j,k}x_j-b_k) $ where the activation function $
\phi $ is a function with bounded value and derivative. These include
single-hidden layer neural networks, polynomials, and sinusoidal models. From a
sample of size $ n $ of possibly noisy values at random sites $ X \in B =
[-1,1]^d $, the minimax mean square error is examined for functions in the
closure of the $ \ell_1 $ hull of ridge functions with activation $ \phi $. It
is shown to be of order $ d/n $ to a fractional power (when $ d $ is of smaller
order than $ n $), and to be of order $ (\log d)/n $ to a fractional power
(when $ d $ is of larger order than $ n $). Dependence on constraints $ v_0 $
and $ v_1 $ on the $ \ell_1 $ norms of inner parameter $ c_0 $ and outer
parameter $ c_1 $, respectively, is also examined. Also, lower and upper bounds
on the fractional power are given. The heart of the analysis is development of
information-theoretic packing numbers for these classes of functions.
