Unsupervised and Supervised Principal Component Analysis: Tutorial
  This is a detailed tutorial paper which explains the Principal Component
Analysis (PCA), Supervised PCA (SPCA), kernel PCA, and kernel SPCA. We start
with projection, PCA with eigen-decomposition, PCA with one and multiple
projection directions, properties of the projection matrix, reconstruction
error minimization, and we connect to auto-encoder. Then, PCA with singular
value decomposition, dual PCA, and kernel PCA are covered. SPCA using both
scoring and Hilbert-Schmidt independence criterion are explained. Kernel SPCA
using both direct and dual approaches are then introduced. We cover all cases
of projection and reconstruction of training and out-of-sample data. Finally,
some simulations are provided on Frey and AT&T face datasets for verifying the
theory in practice.
