Feature selection in machine learning: R\'enyi min-entropy vs Shannon
  entropy
  Feature selection, in the context of machine learning, is the process of
separating the highly predictive feature from those that might be irrelevant or
redundant. Information theory has been recognized as a useful concept for this
task, as the prediction power stems from the correlation, i.e., the mutual
information, between features and labels. Many algorithms for feature selection
in the literature have adopted the Shannon-entropy-based mutual information. In
this paper, we explore the possibility of using R\'enyi min-entropy instead. In
particular, we propose an algorithm based on a notion of conditional R\'enyi
min-entropy that has been recently adopted in the field of security and
privacy, and which is strictly related to the Bayes error. We prove that in
general the two approaches are incomparable, in the sense that we show that we
can construct datasets on which the R\'enyi-based algorithm performs better
than the corresponding Shannon-based one, and datasets on which the situation
is reversed. In practice, however, when considering datasets of real data, it
seems that the R\'enyi-based algorithm tends to outperform the other one. We
have effectuate several experiments on the BASEHOCK, SEMEION, and GISETTE
datasets, and in all of them we have indeed observed that the R\'enyi-based
algorithm gives better results.
