Nonparametric Canonical Correlation Analysis
  Canonical correlation analysis (CCA) is a classical representation learning
technique for finding correlated variables in multi-view data. Several
nonlinear extensions of the original linear CCA have been proposed, including
kernel and deep neural network methods. These approaches seek maximally
correlated projections among families of functions, which the user specifies
(by choosing a kernel or neural network structure), and are computationally
demanding. Interestingly, the theory of nonlinear CCA, without functional
restrictions, had been studied in the population setting by Lancaster already
in the 1950s, but these results have not inspired practical algorithms. We
revisit Lancaster's theory to devise a practical algorithm for nonparametric
CCA (NCCA). Specifically, we show that the solution can be expressed in terms
of the singular value decomposition of a certain operator associated with the
joint density of the views. Thus, by estimating the population density from
data, NCCA reduces to solving an eigenvalue system, superficially like kernel
CCA but, importantly, without requiring the inversion of any kernel matrix. We
also derive a partially linear CCA (PLCCA) variant in which one of the views
undergoes a linear projection while the other is nonparametric. Using a kernel
density estimate based on a small number of nearest neighbors, our NCCA and
PLCCA algorithms are memory-efficient, often run much faster, and perform
better than kernel CCA and comparable to deep CCA.
