Communication-efficient distributed eigenspace estimation
  Distributed computing is a standard way to scale up machine learning and data
science algorithms to process large amounts of data. In such settings, avoiding
communication amongst machines is paramount for achieving high performance.
Rather than distribute the computation of existing algorithms, a common
practice for avoiding communication is to compute local solutions or parameter
estimates on each machine and then combine the results; in many convex
optimization problems, even simple averaging of local solutions can work well.
However, these schemes do not work when the local solutions are not unique.
Spectral methods are a collection of such problems, where solutions are
orthonormal bases of the leading invariant subspace of an associated data
matrix, which are only unique up to rotation and reflections. Here, we develop
a communication-efficient distributed algorithm for computing the leading
invariant subspace of a data matrix. Our algorithm uses a novel alignment
scheme that minimizes the Procrustean distance between local solutions and a
reference solution, and only requires a single round of communication. For the
important case of principal component analysis (PCA), we show that our
algorithm achieves a similar error rate to that of a centralized estimator. We
present numerical experiments demonstrating the efficacy of our proposed
algorithm for distributed PCA, as well as other problems where solutions
exhibit rotational symmetry, such as node embeddings for graph data and
spectral initialization for quadratic sensing.
