Distributed Principal Subspace Analysis for Partitioned Big Data:
  Algorithms, Analysis, and Implementation
  Principal Subspace Analysis (PSA) -- and its sibling, Principal Component
Analysis (PCA) -- is one of the most popular approaches for dimensionality
reduction in signal processing and machine learning. But centralized PSA/PCA
solutions are fast becoming irrelevant in the modern era of big data, in which
the number of samples and/or the dimensionality of samples often exceed the
storage and/or computational capabilities of individual machines. This has led
to the study of distributed PSA/PCA solutions, in which the data are
partitioned across multiple machines and an estimate of the principal subspace
is obtained through collaboration among the machines. It is in this vein that
this paper revisits the problem of distributed PSA/PCA under the general
framework of an arbitrarily connected network of machines that lacks a central
server. The main contributions of the paper in this regard are threefold.
First, two algorithms are proposed in the paper that can be used for
distributed PSA/PCA, with one in the case of data partitioned across samples
and the other in the case of data partitioned across (raw) features. Second, in
the case of sample-wise partitioned data, the proposed algorithm and a variant
of it are analyzed, and their convergence to the true subspace at linear rates
is established. Third, extensive experiments on both synthetic and real-world
data are carried out to validate the usefulness of the proposed algorithms. In
particular, in the case of sample-wise partitioned data, an MPI-based
distributed implementation is carried out to study the interplay between
network topology and communications cost as well as to study the effects of
straggler machines on the proposed algorithms.
