Nonstationary Nonparametric Online Learning: Balancing Dynamic Regret
  and Model Parsimony
  An open challenge in supervised learning is \emph{conceptual drift}: a data
point begins as classified according to one label, but over time the notion of
that label changes. Beyond linear autoregressive models, transfer and meta
learning address drift, but require data that is representative of disparate
domains at the outset of training. To relax this requirement, we propose a
memory-efficient \emph{online} universal function approximator based on
compressed kernel methods. Our approach hinges upon viewing non-stationary
learning as online convex optimization with dynamic comparators, for which
performance is quantified by dynamic regret.
  Prior works control dynamic regret growth only for linear models. In
contrast, we hypothesize actions belong to reproducing kernel Hilbert spaces
(RKHS). We propose a functional variant of online gradient descent (OGD)
operating in tandem with greedy subspace projections. Projections are necessary
to surmount the fact that RKHS functions have complexity proportional to time.
  For this scheme, we establish sublinear dynamic regret growth in terms of
both loss variation and functional path length, and that the memory of the
function sequence remains moderate. Experiments demonstrate the usefulness of
the proposed technique for online nonlinear regression and classification
problems with non-stationary data.
