Deep Feature Selection using a Teacher-Student Network
  High-dimensional data in many machine learning applications leads to
computational and analytical complexities. Feature selection provides an
effective way for solving these problems by removing irrelevant and redundant
features, thus reducing model complexity and improving accuracy and
generalization capability of the model. In this paper, we present a novel
teacher-student feature selection (TSFS) method in which a 'teacher' (a deep
neural network or a complicated dimension reduction method) is first employed
to learn the best representation of data in low dimension. Then a 'student'
network (a simple neural network) is used to perform feature selection by
minimizing the reconstruction error of low dimensional representation. Although
the teacher-student scheme is not new, to the best of our knowledge, it is the
first time that this scheme is employed for feature selection. The proposed
TSFS can be used for both supervised and unsupervised feature selection. This
method is evaluated on different datasets and is compared with state-of-the-art
existing feature selection methods. The results show that TSFS performs better
in terms of classification and clustering accuracies and reconstruction error.
Moreover, experimental evaluations demonstrate a low degree of sensitivity to
parameter selection in the proposed method.
