Stochastic Gradient Descent, Weighted Sampling, and the Randomized
  Kaczmarz algorithm
  We obtain an improved finite-sample guarantee on the linear convergence of
stochastic gradient descent for smooth and strongly convex objectives,
improving from a quadratic dependence on the conditioning $(L/\mu)^2$ (where
$L$ is a bound on the smoothness and $\mu$ on the strong convexity) to a linear
dependence on $L/\mu$. Furthermore, we show how reweighting the sampling
distribution (i.e. importance sampling) is necessary in order to further
improve convergence, and obtain a linear dependence in the average smoothness,
dominating previous results. We also discuss importance sampling for SGD more
broadly and show how it can improve convergence also in other scenarios. Our
results are based on a connection we make between SGD and the randomized
Kaczmarz algorithm, which allows us to transfer ideas between the separate
bodies of literature studying each of the two methods. In particular, we recast
the randomized Kaczmarz algorithm as an instance of SGD, and apply our results
to prove its exponential convergence, but to the solution of a weighted least
squares problem rather than the original least squares problem. We then present
a modified Kaczmarz algorithm with partially biased sampling which does
converge to the original least squares solution with the same exponential
convergence rate.
