Fast Sketching of Polynomial Kernels of Polynomial Degree
  Kernel methods are fundamental in machine learning, and faster algorithms for
kernel approximation provide direct speedups for many core tasks in machine
learning. The polynomial kernel is especially important as other kernels can
often be approximated by the polynomial kernel via a Taylor series expansion.
Recent techniques in oblivious sketching reduce the dependence in the running
time on the degree $q$ of the polynomial kernel from exponential to polynomial,
which is useful for the Gaussian kernel, for which $q$ can be chosen to be
polylogarithmic. However, for more slowly growing kernels, such as the neural
tangent and arc-cosine kernels, $q$ needs to be polynomial, and previous work
incurs a polynomial factor slowdown in the running time. We give a new
oblivious sketch which greatly improves upon this running time, by removing the
dependence on $q$ in the leading order term. Combined with a novel sampling
scheme, we give the fastest algorithms for approximating a large family of
slow-growing kernels.
