Tight Convergence Rate Bounds for Optimization Under Power Law Spectral
  Conditions
  Performance of optimization on quadratic problems sensitively depends on the
low-lying part of the spectrum. For large (effectively infinite-dimensional)
problems, this part of the spectrum can often be naturally represented or
approximated by power law distributions. In this paper we perform a systematic
study of a range of classical single-step and multi-step first order
optimization algorithms, with adaptive and non-adaptive, constant and
non-constant learning rates: vanilla Gradient Descent, Steepest Descent, Heavy
Ball, and Conjugate Gradients. For each of these, we prove that a power law
spectral assumption entails a power law for convergence rate of the algorithm,
with the convergence rate exponent given by a specific multiple of the spectral
exponent. We establish both upper and lower bounds, showing that the results
are tight. Finally, we demonstrate applications of these results to kernel
learning and training of neural networks in the NTK regime.
