How Good is SGD with Random Shuffling?
  We study the performance of stochastic gradient descent (SGD) on smooth and
strongly-convex finite-sum optimization problems. In contrast to the majority
of existing theoretical works, which assume that individual functions are
sampled with replacement, we focus here on popular but poorly-understood
heuristics, which involve going over random permutations of the individual
functions. This setting has been investigated in several recent works, but the
optimal error rates remain unclear. In this paper, we provide lower bounds on
the expected optimization error with these heuristics (using SGD with any
constant step size), which elucidate their advantages and disadvantages. In
particular, we prove that after $k$ passes over $n$ individual functions, if
the functions are re-shuffled after every pass, the best possible optimization
error for SGD is at least $\Omega\left(1/(nk)^2+1/nk^3\right)$, which partially
corresponds to recently derived upper bounds. Moreover, if the functions are
only shuffled once, then the lower bound increases to $\Omega(1/nk^2)$. Since
there are strictly smaller upper bounds for repeated reshuffling, this proves
an inherent performance gap between SGD with single shuffling and repeated
shuffling. As a more minor contribution, we also provide a non-asymptotic
$\Omega(1/k^2)$ lower bound (independent of $n$) for the incremental gradient
method, when no random shuffling takes place. Finally, we provide an indication
that our lower bounds are tight, by proving matching upper bounds for
univariate quadratic functions.
