Concentration of Random Feature Matrices in High-Dimensions
  The spectra of random feature matrices provide essential information on the
conditioning of the linear system used in random feature regression problems
and are thus connected to the consistency and generalization of random feature
models. Random feature matrices are asymmetric rectangular nonlinear matrices
depending on two input variables, the data and the weights, which can make
their characterization challenging. We consider two settings for the two input
variables, either both are random variables or one is a random variable and the
other is well-separated, i.e. there is a minimum distance between points. With
conditions on the dimension, the complexity ratio, and the sampling variance,
we show that the singular values of these matrices concentrate near their full
expectation and near one with high-probability. In particular, since the
dimension depends only on the logarithm of the number of random weights or the
number of data points, our complexity bounds can be achieved even in moderate
dimensions for many practical setting. The theoretical results are verified
with numerical experiments.
