Deformed semicircle law and concentration of nonlinear random matrices
  for ultra-wide neural networks
  In this paper, we study the two-layer fully connected neural network given by
$f(X)=\frac{1}{\sqrt{d_1}}\boldsymbol{a}^\top\sigma\left(WX\right)$, where
$X\in\mathbb{R}^{d_0\times n}$ is a deterministic data matrix,
$W\in\mathbb{R}^{d_1\times d_0}$ and $\boldsymbol{a}\in\mathbb{R}^{d_1}$ are
random Gaussian weights, and $\sigma$ is a nonlinear activation function. We
obtain the limiting spectral distributions of two kernel matrices related to
$f(X)$: the empirical conjugate kernel (CK) and neural tangent kernel (NTK),
beyond the linear-width regime ($d_1\asymp n$). Under the ultra-width regime
$d_1/n\to\infty$, with proper assumptions on $X$ and $\sigma$, a deformed
semicircle law appears. Such limiting law is first proved for general centered
sample covariance matrices with correlation and then specified for our neural
network model. We also prove non-asymptotic concentrations of empirical CK and
NTK around their limiting kernel in the spectral norm, and lower bounds on
their smallest eigenvalues. As an application, we verify the random feature
regression achieves the same asymptotic performance as its limiting kernel
regression in ultra-width limit. The limiting training and test errors for
random feature regression are calculated by corresponding kernel regression. We
also provide a nonlinear Hanson-Wright inequality suitable for neural networks
with random weights and Lipschitz activation functions.
