MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for
  scaling Deep Learning
  Existing Deep Learning frameworks exclusively use either Parameter Server(PS)
approach or MPI parallelism. In this paper, we discuss the drawbacks of such
approaches and propose a generic framework supporting both PS and MPI
programming paradigms, co-existing at the same time. The key advantage of the
new model is to embed the scaling benefits of MPI parallelism into the loosely
coupled PS task model. Apart from providing a practical usage model of MPI in
cloud, such framework allows for novel communication avoiding algorithms that
do parameter averaging in Stochastic Gradient Descent(SGD) approaches. We show
how MPI and PS models can synergestically apply algorithms such as Elastic SGD
to improve the rate of convergence against existing approaches. These new
algorithms directly help scaling SGD clusterwide. Further, we also optimize the
critical component of the framework, namely global aggregation or allreduce
using a novel concept of tensor collectives. These treat a group of vectors on
a node as a single object allowing for the existing single vector algorithms to
be directly applicable. We back our claims with sufficient emperical evidence
using large scale ImageNet 1K data. Our framework is built upon MXNET but the
design is generic and can be adapted to other popular DL infrastructures.
