Randomized Block Krylov Methods for Stronger and Faster Approximate
  Singular Value Decomposition
  Since being analyzed by Rokhlin, Szlam, and Tygert and popularized by Halko,
Martinsson, and Tropp, randomized Simultaneous Power Iteration has become the
method of choice for approximate singular value decomposition. It is more
accurate than simpler sketching algorithms, yet still converges quickly for any
matrix, independently of singular value gaps. After $\tilde{O}(1/\epsilon)$
iterations, it gives a low-rank approximation within $(1+\epsilon)$ of optimal
for spectral norm error.
  We give the first provable runtime improvement on Simultaneous Iteration: a
simple randomized block Krylov method, closely related to the classic Block
Lanczos algorithm, gives the same guarantees in just
$\tilde{O}(1/\sqrt{\epsilon})$ iterations and performs substantially better
experimentally. Despite their long history, our analysis is the first of a
Krylov subspace method that does not depend on singular value gaps, which are
unreliable in practice.
  Furthermore, while it is a simple accuracy benchmark, even $(1+\epsilon)$
error for spectral norm low-rank approximation does not imply that an algorithm
returns high quality principal components, a major issue for data applications.
We address this problem for the first time by showing that both Block Krylov
Iteration and a minor modification of Simultaneous Iteration give nearly
optimal PCA for any matrix. This result further justifies their strength over
non-iterative sketching methods.
  Finally, we give insight beyond the worst case, justifying why both
algorithms can run much faster in practice than predicted. We clarify how
simple techniques can take advantage of common matrix properties to
significantly improve runtime.
