Online Supervised Subspace Tracking
  We present a framework for supervised subspace tracking, when there are two
time series $x_t$ and $y_t$, one being the high-dimensional predictors and the
other being the response variables and the subspace tracking needs to take into
consideration of both sequences. It extends the classic online subspace
tracking work which can be viewed as tracking of $x_t$ only. Our online
sufficient dimensionality reduction (OSDR) is a meta-algorithm that can be
applied to various cases including linear regression, logistic regression,
multiple linear regression, multinomial logistic regression, support vector
machine, the random dot product model and the multi-scale union-of-subspace
model. OSDR reduces data-dimensionality on-the-fly with low-computational
complexity and it can also handle missing data and dynamic data. OSDR uses an
alternating minimization scheme and updates the subspace via gradient descent
on the Grassmannian manifold. The subspace update can be performed efficiently
utilizing the fact that the Grassmannian gradient with respect to the subspace
in many settings is rank-one (or low-rank in certain cases). The optimization
problem for OSDR is non-convex and hard to analyze in general; we provide
convergence analysis of OSDR in a simple linear regression setting. The good
performance of OSDR compared with the conventional unsupervised subspace
tracking are demonstrated via numerical examples on simulated and real data.
