Communications Inspired Linear Discriminant Analysis
  We study the problem of supervised linear dimensionality reduction, taking an
information-theoretic viewpoint. The linear projection matrix is designed by
maximizing the mutual information between the projected signal and the class
label (based on a Shannon entropy measure). By harnessing a recent theoretical
result on the gradient of mutual information, the above optimization problem
can be solved directly using gradient descent, without requiring simplification
of the objective function. Theoretical analysis and empirical comparison are
made between the proposed method and two closely related methods (Linear
Discriminant Analysis and Information Discriminant Analysis), and comparisons
are also made with a method in which Renyi entropy is used to define the mutual
information (in this case the gradient may be computed simply, under a special
parameter setting). Relative to these alternative approaches, the proposed
method achieves promising results on real datasets.
