Improved Convergence Rates for Sparse Approximation Methods in
  Kernel-Based Learning
  Kernel-based models such as kernel ridge regression and Gaussian processes
are ubiquitous in machine learning applications for regression and
optimization. It is well known that a serious downside for kernel-based models
is the high computational cost; given a dataset of $n$ samples, the cost grows
as $\mathcal{O}(n^3)$. Existing sparse approximation methods can yield a
significant reduction in the computational cost, effectively reducing the real
world cost down to as low as $\mathcal{O}(n)$ in certain cases. Despite this
remarkable empirical success, significant gaps remain in the existing results
for the analytical confidence bounds on the error due to approximation. In this
work, we provide novel confidence intervals for the Nystr\"om method and the
sparse variational Gaussian processes approximation method. Our confidence
intervals lead to improved error bounds in both regression and optimization. We
establish these confidence intervals using novel interpretations of the
approximate (surrogate) posterior variance of the models.
