Understanding and Optimizing the Performance of Distributed Machine
  Learning Applications on Apache Spark
  In this paper we explore the performance limits of Apache Spark for machine
learning applications. We begin by analyzing the characteristics of a
state-of-the-art distributed machine learning algorithm implemented in Spark
and compare it to an equivalent reference implementation using the high
performance computing framework MPI. We identify critical bottlenecks of the
Spark framework and carefully study their implications on the performance of
the algorithm. In order to improve Spark performance we then propose a number
of practical techniques to alleviate some of its overheads. However, optimizing
computational efficiency and framework related overheads is not the only key to
performance -- we demonstrate that in order to get the best performance out of
any implementation it is necessary to carefully tune the algorithm to the
respective trade-off between computation time and communication latency. The
optimal trade-off depends on both the properties of the distributed algorithm
as well as infrastructure and framework-related characteristics. Finally, we
apply these technical and algorithmic optimizations to three different
distributed linear machine learning algorithms that have been implemented in
Spark. We present results using five large datasets and demonstrate that by
using the proposed optimizations, we can achieve a reduction in the performance
difference between Spark and MPI from 20x to 2x.
