Consistent Online Gaussian Process Regression Without the Sample
  Complexity Bottleneck
  Gaussian processes provide a framework for nonlinear nonparametric Bayesian
inference widely applicable across science and engineering. Unfortunately,
their computational burden scales cubically with the training sample size,
which in the case that samples arrive in perpetuity, approaches infinity. This
issue necessitates approximations for use with streaming data, which to date
mostly lack convergence guarantees. Thus, we develop the first online Gaussian
process approximation that preserves convergence to the population posterior,
i.e., asymptotic posterior consistency, while ameliorating its intractable
complexity growth with the sample size. We propose an online compression scheme
that, following each a posteriori update, fixes an error neighborhood with
respect to the Hellinger metric centered at the current posterior, and greedily
tosses out past kernel dictionary elements until its boundary is hit. We call
the resulting method Parsimonious Online Gaussian Processes (POG). For
diminishing error radius, exact asymptotic consistency is preserved (Theorem
1(i)) at the cost of unbounded memory in the limit. On the other hand, for
constant error radius, POG converges to a neighborhood of the population
posterior (Theorem 1(ii))but with finite memory at-worst determined by the
metric entropy of the feature space (Theorem 2). Experimental results are
presented on several nonlinear regression problems which illuminates the merits
of this approach as compared with alternatives that fix the subspace dimension
defining the history of past points.
