Covariance-free Partial Least Squares: An Incremental Dimensionality
  Reduction Method
  Dimensionality reduction plays an important role in computer vision problems
since it reduces computational cost and is often capable of yielding more
discriminative data representation. In this context, Partial Least Squares
(PLS) has presented notable results in tasks such as image classification and
neural network optimization. However, PLS is infeasible on large datasets, such
as ImageNet, because it requires all the data to be in memory in advance, which
is often impractical due to hardware limitations. Additionally, this
requirement prevents us from employing PLS on streaming applications where the
data are being continuously generated. Motivated by this, we propose a novel
incremental PLS, named Covariance-free Incremental Partial Least Squares
(CIPLS), which learns a low-dimensional representation of the data using a
single sample at a time. In contrast to other state-of-the-art approaches,
instead of adopting a partially-discriminative or SGD-based model, we extend
Nonlinear Iterative Partial Least Squares (NIPALS) -- the standard algorithm
used to compute PLS -- for incremental processing. Among the advantages of this
approach are the preservation of discriminative information across all
components, the possibility of employing its score matrices for feature
selection, and its computational efficiency. We validate CIPLS on face
verification and image classification tasks, where it outperforms several other
incremental dimensionality reduction techniques. In the context of feature
selection, CIPLS achieves comparable results when compared to state-of-the-art
techniques.
