An Information-theoretic Approach to Unsupervised Feature Selection for
  High-Dimensional Data
  In this paper, we propose an information-theoretic approach to design the
functional representations to extract the hidden common structure shared by a
set of random variables. The main idea is to measure the common information
between the random variables by Watanabe's total correlation, and then find the
hidden attributes of these random variables such that the common information is
reduced the most given these attributes. We show that these attributes can be
characterized by an exponential family specified by the eigen-decomposition of
some pairwise joint distribution matrix. Then, we adopt the log-likelihood
functions for estimating these attributes as the desired functional
representations of the random variables, and show that such representations are
informative to describe the common structure. Moreover, we design both the
multivariate alternating conditional expectation (MACE) algorithm to compute
the proposed functional representations for discrete data, and a novel neural
network training approach for continuous or high-dimensional data. Furthermore,
we show that our approach has deep connections to existing techniques, such as
Hirschfeld-Gebelein-R\'{e}nyi (HGR) maximal correlation, linear principal
component analysis (PCA), and consistent functional map, which establishes
insightful connections between information theory and machine learning.
Finally, the performances of our algorithms are validated by numerical
simulations.
