Feature selection with optimal coordinate ascent (OCA)
  In machine learning, Feature Selection (FS) is a major part of efficient
algorithm. It fuels the algorithm and is the starting block for our prediction.
In this paper, we present a new method, called Optimal Coordinate Ascent (OCA)
that allows us selecting features among block and individual features. OCA
relies on coordinate ascent to find an optimal solution for gradient boosting
methods score (number of correctly classified samples). OCA takes into account
the notion of dependencies between variables forming blocks in our
optimization. The coordinate ascent optimization solves the issue of the NP
hard original problem where the number of combinations rapidly explode making a
grid search unfeasible. It reduces considerably the number of iterations
changing this NP hard problem into a polynomial search one. OCA brings
substantial differences and improvements compared to previous coordinate ascent
feature selection method: we group variables into block and individual
variables instead of a binary selection. Our initial guess is based on the
k-best group variables making our initial point more robust. We also introduced
new stopping criteria making our optimization faster. We compare these two
methods on our data set. We found that our method outperforms the initial one.
We also compare our method to the Recursive Feature Elimination (RFE) method
and find that OCA leads to the minimum feature set with the highest score. This
is a nice byproduct of our method as it provides empirically the most compact
data set with optimal performance.
