A phase transition for finding needles in nonlinear haystacks with LASSO
  artificial neural networks
  To fit sparse linear associations, a LASSO sparsity inducing penalty with a
single hyperparameter provably allows to recover the important features
(needles) with high probability in certain regimes even if the sample size is
smaller than the dimension of the input vector (haystack). More recently
learners known as artificial neural networks (ANN) have shown great successes
in many machine learning tasks, in particular fitting nonlinear associations.
Small learning rate, stochastic gradient descent algorithm and large training
set help to cope with the explosion in the number of parameters present in deep
neural networks. Yet few ANN learners have been developed and studied to find
needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN
learner, like for sparse linear associations, exhibits a phase transition in
the probability of retrieving the needles, which we do not observe with other
ANN learners. To select our penalty parameter, we generalize the universal
threshold of Donoho and Johnstone (1994) which is a better rule than the
conservative (too many false detections) and expensive cross-validation. In the
spirit of simulated annealing, we propose a warm-start sparsity inducing
algorithm to solve the high-dimensional, non-convex and non-differentiable
optimization problem. We perform precise Monte Carlo simulations to show the
effectiveness of our approach.
