On Universal Features for High-Dimensional Learning and Inference
  We consider the problem of identifying universal low-dimensional features
from high-dimensional data for inference tasks in settings involving learning.
For such problems, we introduce natural notions of universality and we show a
local equivalence among them. Our analysis is naturally expressed via
information geometry, and represents a conceptually and computationally useful
analysis. The development reveals the complementary roles of the singular value
decomposition, Hirschfeld-Gebelein-R\'enyi maximal correlation, the canonical
correlation and principle component analyses of Hotelling and Pearson, Tishby's
information bottleneck, Wyner's common information, Ky Fan $k$-norms, and
Brieman and Friedman's alternating conditional expectations algorithm. We
further illustrate how this framework facilitates understanding and optimizing
aspects of learning systems, including multinomial logistic (softmax)
regression and the associated neural network architecture, matrix factorization
methods for collaborative filtering and other applications, rank-constrained
multivariate linear regression, and forms of semi-supervised learning.
