Stochastic Gradient Descent in Hilbert Scales: Smoothness,
  Preconditioning and Earlier Stopping
  Stochastic Gradient Descent (SGD) has become the method of choice for solving
a broad range of machine learning problems. However, some of its learning
properties are still not fully understood. We consider least squares learning
in reproducing kernel Hilbert spaces (RKHSs) and extend the classical SGD
analysis to a learning setting in Hilbert scales, including Sobolev spaces and
Diffusion spaces on compact Riemannian manifolds. We show that even for
well-specified models, violation of a traditional benchmark smoothness
assumption has a tremendous effect on the learning rate. In addition, we show
that for miss-specified models, preconditioning in an appropriate Hilbert scale
helps to reduce the number of iterations, i.e. allowing for "earlier stopping".
