Generalization Bounds for Sparse Random Feature Expansions
  Random feature methods have been successful in various machine learning
tasks, are easy to compute, and come with theoretical accuracy bounds. They
serve as an alternative approach to standard neural networks since they can
represent similar function spaces without a costly training phase. However, for
accuracy, random feature methods require more measurements than trainable
parameters, limiting their use for data-scarce applications or problems in
scientific machine learning. This paper introduces the sparse random feature
expansion to obtain parsimonious random feature models. Specifically, we
leverage ideas from compressive sensing to generate random feature expansions
with theoretical guarantees even in the data-scarce setting. In particular, we
provide generalization bounds for functions in a certain class (that is dense
in a reproducing kernel Hilbert space) depending on the number of samples and
the distribution of features. The generalization bounds improve with additional
structural conditions, such as coordinate sparsity, compact clusters of the
spectrum, or rapid spectral decay. In particular, by introducing sparse
features, i.e. features with random sparse weights, we provide improved bounds
for low order functions. We show that the sparse random feature expansions
outperforms shallow networks in several scientific machine learning tasks.
