Scalable Manifold Learning for Big Data with Apache Spark
  Non-linear spectral dimensionality reduction methods, such as Isomap, remain
important technique for learning manifolds. However, due to computational
complexity, exact manifold learning using Isomap is currently impossible from
large-scale data. In this paper, we propose a distributed memory framework
implementing end-to-end exact Isomap under Apache Spark model. We show how each
critical step of the Isomap algorithm can be efficiently realized using basic
Spark model, without the need to provision data in the secondary storage. We
show how the entire method can be implemented using PySpark, offloading compute
intensive linear algebra routines to BLAS. Through experimental results, we
demonstrate excellent scalability of our method, and we show that it can
process datasets orders of magnitude larger than what is currently possible,
using a 25-node parallel~cluster.
