Theoretical Connection between Locally Linear Embedding, Factor
  Analysis, and Probabilistic PCA
  Locally Linear Embedding (LLE) is a nonlinear spectral dimensionality
reduction and manifold learning method. It has two main steps which are linear
reconstruction and linear embedding of points in the input space and embedding
space, respectively. In this work, we look at the linear reconstruction step
from a stochastic perspective where it is assumed that every data point is
conditioned on its linear reconstruction weights as latent factors. The
stochastic linear reconstruction of LLE is solved using expectation
maximization. We show that there is a theoretical connection between three
fundamental dimensionality reduction methods, i.e., LLE, factor analysis, and
probabilistic Principal Component Analysis (PCA). The stochastic linear
reconstruction of LLE is formulated similar to the factor analysis and
probabilistic PCA. It is also explained why factor analysis and probabilistic
PCA are linear and LLE is a nonlinear method. This work combines and makes a
bridge between two broad approaches of dimensionality reduction, i.e., the
spectral and probabilistic algorithms.
