Asymptotic Network Independence in Distributed Stochastic Optimization
  for Machine Learning
  We provide a discussion of several recent results which, in certain
scenarios, are able to overcome a barrier in distributed stochastic
optimization for machine learning. Our focus is the so-called asymptotic
network independence property, which is achieved whenever a distributed method
executed over a network of n nodes asymptotically converges to the optimal
solution at a comparable rate to a centralized method with the same
computational power as the entire network. We explain this property through an
example involving the training of ML models and sketch a short mathematical
analysis for comparing the performance of distributed stochastic gradient
descent (DSGD) with centralized stochastic gradient decent (SGD).
