Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
  We design and implement a distributed multinode synchronous SGD algorithm,
without altering hyper parameters, or compressing data, or altering algorithmic
behavior. We perform a detailed analysis of scaling, and identify optimal
design points for different networks. We demonstrate scaling of CNNs on 100s of
nodes, and present what we believe to be record training throughputs. A 512
minibatch VGG-A CNN training run is scaled 90X on 128 nodes. Also 256 minibatch
VGG-A and OverFeat-FAST networks are scaled 53X and 42X respectively on a 64
node cluster. We also demonstrate the generality of our approach via
best-in-class 6.5X scaling for a 7-layer DNN on 16 nodes. Thereafter we attempt
to democratize deep-learning by training on an Ethernet based AWS cluster and
show ~14X scaling on 16 nodes.
