Elastic Bulk Synchronous Parallel Model for Distributed Deep Learning
  The bulk synchronous parallel (BSP) is a celebrated synchronization model for
general-purpose parallel computing that has successfully been employed for
distributed training of machine learning models. A prevalent shortcoming of the
BSP is that it requires workers to wait for the straggler at every iteration.
To ameliorate this shortcoming of classic BSP, we propose ELASTICBSP a model
that aims to relax its strict synchronization requirement. The proposed model
offers more flexibility and adaptability during the training phase, without
sacrificing on the accuracy of the trained model. We also propose an efficient
method that materializes the model, named ZIPLINE. The algorithm is tunable and
can effectively balance the trade-off between quality of convergence and
iteration throughput, in order to accommodate different environments or
applications. A thorough experimental evaluation demonstrates that our proposed
ELASTICBSP model converges faster and to a higher accuracy than the classic
BSP. It also achieves comparable (if not higher) accuracy than the other
sensible synchronization models.
