Scaling the Wild: Decentralizing Hogwild!-style Shared-memory SGD
  Powered by the simplicity of lock-free asynchrony, Hogwilld! is a go-to
approach to parallelize SGD over a shared-memory setting. Despite its
popularity and concomitant extensions, such as PASSM+ wherein concurrent
processes update a shared model with partitioned gradients, scaling it to
decentralized workers has surprisingly been relatively unexplored. To our
knowledge, there is no convergence theory of such methods, nor systematic
numerical comparisons evaluating speed-up.
  In this paper, we propose an algorithm incorporating decentralized
distributed memory computing architecture with each node running
multiprocessing parallel shared-memory SGD itself. Our scheme is based on the
following algorithmic tools and features: (a) asynchronous local gradient
updates on the shared-memory of workers, (b) partial backpropagation, and (c)
non-blocking in-place averaging of the local models. We prove that our method
guarantees ergodic convergence rates for non-convex objectives. On the
practical side, we show that the proposed method exhibits improved throughput
and competitive accuracy for standard image classification benchmarks on the
CIFAR-10, CIFAR-100, and Imagenet datasets. Our code is available at
https://github.com/bapi/LPP-SGD.
