Accumulations of Projections--A Unified Framework for Random Sketches in
  Kernel Ridge Regression
  Building a sketch of an n-by-n empirical kernel matrix is a common approach
to accelerate the computation of many kernel methods. In this paper, we propose
a unified framework of constructing sketching methods in kernel ridge
regression (KRR), which views the sketching matrix S as an accumulation of m
rescaled sub-sampling matrices with independent columns. Our framework
incorporates two commonly used sketching methods, sub-sampling sketches (known
as the Nystr\"om method) and sub-Gaussian sketches, as special cases with m=1
and m=infinity respectively. Under the new framework, we provide a unified
error analysis of sketching approximation and show that our accumulation scheme
improves the low accuracy of sub-sampling sketches when certain incoherence
characteristic is high, and accelerates the more accurate but computationally
heavier sub-Gaussian sketches. By optimally choosing the number m of
accumulations, we show that a best trade-off between computational efficiency
and statistical accuracy can be achieved. In practice, the sketching method can
be as efficiently implemented as the sub-sampling sketches, as only minor extra
matrix additions are needed. Our empirical evaluations also demonstrate that
the proposed method may attain the accuracy close to sub-Gaussian sketches,
while is as efficient as sub-sampling-based sketches.
