What needles do sparse neural networks find in nonlinear haystacks
  Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids
over-fitting, especially in situations where noise is high and the training set
is small in comparison to the number of features. For linear models, such an
approach provably also recovers the important features with high probability in
regimes for a well-chosen penalty parameter. The typical way of setting the
penalty parameter is by splitting the data set and performing the
cross-validation, which is (1) computationally expensive and (2) not desirable
when the data set is already small to be further split (for example,
whole-genome sequence data). In this study, we establish the theoretical
foundation to select the penalty parameter without cross-validation based on
bounding with a high probability the infinite norm of the gradient of the loss
function at zero under the zero-feature assumption. Our approach is a
generalization of the universal threshold of Donoho and Johnstone (1994) to
nonlinear ANN learning. We perform a set of comprehensive Monte Carlo
simulations on a simple model, and the numerical results show the effectiveness
of the proposed approach.
