Discrete R\'enyi Classifiers
  Consider the binary classification problem of predicting a target variable
$Y$ from a discrete feature vector $X = (X_1,...,X_d)$. When the probability
distribution $\mathbb{P}(X,Y)$ is known, the optimal classifier, leading to the
minimum misclassification rate, is given by the Maximum A-posteriori
Probability decision rule. However, estimating the complete joint distribution
$\mathbb{P}(X,Y)$ is computationally and statistically impossible for large
values of $d$. An alternative approach is to first estimate some low order
marginals of $\mathbb{P}(X,Y)$ and then design the classifier based on the
estimated low order marginals. This approach is also helpful when the complete
training data instances are not available due to privacy concerns. In this
work, we consider the problem of finding the optimum classifier based on some
estimated low order marginals of $(X,Y)$. We prove that for a given set of
marginals, the minimum Hirschfeld-Gebelein-Renyi (HGR) correlation principle
introduced in [1] leads to a randomized classification rule which is shown to
have a misclassification rate no larger than twice the misclassification rate
of the optimal classifier. Then, under a separability condition, we show that
the proposed algorithm is equivalent to a randomized linear regression
approach. In addition, this method naturally results in a robust feature
selection method selecting a subset of features having the maximum worst case
HGR correlation with the target variable. Our theoretical upper-bound is
similar to the recent Discrete Chebyshev Classifier (DCC) approach [2], while
the proposed algorithm has significant computational advantages since it only
requires solving a least square optimization problem. Finally, we numerically
compare our proposed algorithm with the DCC classifier and show that the
proposed algorithm results in better misclassification rate over various
datasets.
