A Data and Model-Parallel, Distributed and Scalable Framework for
  Training of Deep Networks in Apache Spark
  Training deep networks is expensive and time-consuming with the training
period increasing with data size and growth in model parameters. In this paper,
we provide a framework for distributed training of deep networks over a cluster
of CPUs in Apache Spark. The framework implements both Data Parallelism and
Model Parallelism making it suitable to use for deep networks which require
huge training data and model parameters which are too big to fit into the
memory of a single machine. It can be scaled easily over a cluster of cheap
commodity hardware to attain significant speedup and obtain better results
making it quite economical as compared to farm of GPUs and supercomputers. We
have proposed a new algorithm for training of deep networks for the case when
the network is partitioned across the machines (Model Parallelism) along with
detailed cost analysis and proof of convergence of the same. We have developed
implementations for Fully-Connected Feedforward Networks, Convolutional Neural
Networks, Recurrent Neural Networks and Long Short-Term Memory architectures.
We present the results of extensive simulations demonstrating the speedup and
accuracy obtained by our framework for different sizes of the data and model
parameters with variation in the number of worker cores/partitions; thereby
showing that our proposed framework can achieve significant speedup (upto 11X
for CNN) and is also quite scalable.
