Diving into the shallows: a computational perspective on large-scale
  shallow learning
  In this paper we first identify a basic limitation in gradient descent-based
optimization methods when used in conjunctions with smooth kernels. An analysis
based on the spectral properties of the kernel demonstrates that only a
vanishingly small portion of the function space is reachable after a polynomial
number of gradient descent iterations. This lack of approximating power
drastically limits gradient descent for a fixed computational budget leading to
serious over-regularization/underfitting. The issue is purely algorithmic,
persisting even in the limit of infinite data.
  To address this shortcoming in practice, we introduce EigenPro iteration,
based on a preconditioning scheme using a small number of approximately
computed eigenvectors. It can also be viewed as learning a new kernel optimized
for gradient descent. It turns out that injecting this small (computationally
inexpensive and SGD-compatible) amount of approximate second-order information
leads to major improvements in convergence. For large data, this translates
into significant performance boost over the standard kernel methods. In
particular, we are able to consistently match or improve the state-of-the-art
results recently reported in the literature with a small fraction of their
computational budget.
  Finally, we feel that these results show a need for a broader computational
perspective on modern large-scale learning to complement more traditional
statistical and convergence analyses. In particular, many phenomena of
large-scale high-dimensional inference are best understood in terms of
optimization on infinite dimensional Hilbert spaces, where standard algorithms
can sometimes have properties at odds with finite-dimensional intuition. A
systematic analysis concentrating on the approximation power of such algorithms
within a budget of computation may lead to progress both in theory and
practice.
