Approximation beats concentration? An approximation view on inference
  with smooth radial kernels
  Positive definite kernels and their associated Reproducing Kernel Hilbert
Spaces provide a mathematically compelling and practically competitive
framework for learning from data.
  In this paper we take the approximation theory point of view to explore
various aspects of smooth kernels related to their inferential properties. We
analyze eigenvalue decay of kernels operators and matrices, properties of
eigenfunctions/eigenvectors and "Fourier" coefficients of functions in the
kernel space restricted to a discrete set of data points. We also investigate
the fitting capacity of kernels, giving explicit bounds on the fat shattering
dimension of the balls in Reproducing Kernel Hilbert spaces. Interestingly, the
same properties that make kernels very effective approximators for functions in
their "native" kernel space, also limit their capacity to represent arbitrary
functions. We discuss various implications, including those for gradient
descent type methods.
  It is important to note that most of our bounds are measure independent.
Moreover, at least in moderate dimension, the bounds for eigenvalues are much
tighter than the bounds which can be obtained from the usual matrix
concentration results. For example, we see that the eigenvalues of kernel
matrices show nearly exponential decay with constants depending only on the
kernel and the domain. We call this "approximation beats concentration"
phenomenon as even when the data are sampled from a probability distribution,
some of their aspects are better understood in terms of approximation theory.
