Bayesian feature selection with strongly-regularizing priors maps to the
  Ising Model
  Identifying small subsets of features that are relevant for prediction and/or
classification tasks is a central problem in machine learning and statistics.
The feature selection task is especially important, and computationally
difficult, for modern datasets where the number of features can be comparable
to, or even exceed, the number of samples. Here, we show that feature selection
with Bayesian inference takes a universal form and reduces to calculating the
magnetizations of an Ising model, under some mild conditions. Our results
exploit the observation that the evidence takes a universal form for
strongly-regularizing priors --- priors that have a large effect on the
posterior probability even in the infinite data limit. We derive explicit
expressions for feature selection for generalized linear models, a large class
of statistical techniques that include linear and logistic regression. We
illustrate the power of our approach by analyzing feature selection in a
logistic regression-based classifier trained to distinguish between the letters
B and D in the notMNIST dataset.
