Distributed Estimation for Principal Component Analysis: an Enlarged
  Eigenspace Analysis
  The growing size of modern data sets brings many challenges to the existing
statistical estimation approaches, which calls for new distributed
methodologies. This paper studies distributed estimation for a fundamental
statistical machine learning problem, principal component analysis (PCA).
Despite the massive literature on top eigenvector estimation, much less is
presented for the top-$L$-dim ($L>1$) eigenspace estimation, especially in a
distributed manner. We propose a novel multi-round algorithm for constructing
top-$L$-dim eigenspace for distributed data. Our algorithm takes advantage of
shift-and-invert preconditioning and convex optimization. Our estimator is
communication-efficient and achieves a fast convergence rate. In contrast to
the existing divide-and-conquer algorithm, our approach has no restriction on
the number of machines. Theoretically, the traditional Davis-Kahan theorem
requires the explicit eigengap assumption to estimate the top-$L$-dim
eigenspace. To abandon this eigengap assumption, we consider a new route in our
analysis: instead of exactly identifying the top-$L$-dim eigenspace, we show
that our estimator is able to cover the targeted top-$L$-dim population
eigenspace. Our distributed algorithm can be applied to a wide range of
statistical problems based on PCA, such as principal component regression and
single index model. Finally, We provide simulation studies to demonstrate the
performance of the proposed distributed estimator.
