AFS: An Attention-based mechanism for Supervised Feature Selection
  As an effective data preprocessing step, feature selection has shown its
effectiveness to prepare high-dimensional data for many machine learning tasks.
The proliferation of high di-mension and huge volume big data, however, has
brought major challenges, e.g. computation complexity and stability on noisy
data, upon existing feature-selection techniques. This paper introduces a novel
neural network-based feature selection architecture, dubbed Attention-based
Feature Selec-tion (AFS). AFS consists of two detachable modules: an at-tention
module for feature weight generation and a learning module for the problem
modeling. The attention module for-mulates correlation problem among features
and supervision target into a binary classification problem, supported by a
shallow attention net for each feature. Feature weights are generated based on
the distribution of respective feature se-lection patterns adjusted by
backpropagation during the train-ing process. The detachable structure allows
existing off-the-shelf models to be directly reused, which allows for much less
training time, demands for the training data and requirements for expertise. A
hybrid initialization method is also intro-duced to boost the selection
accuracy for datasets without enough samples for feature weight generation.
Experimental results show that AFS achieves the best accuracy and stability in
comparison to several state-of-art feature selection algo-rithms upon both
MNIST, noisy MNIST and several datasets with small samples.
