Decentralized Stochastic Gradient Tracking for Non-convex Empirical Risk
  Minimization
  This paper studies a decentralized stochastic gradient tracking (DSGT)
algorithm for non-convex empirical risk minimization problems over a
peer-to-peer network of nodes, which is in sharp contrast to the existing DSGT
only for convex problems. To ensure exact convergence and handle the variance
among decentralized datasets, each node performs a stochastic gradient (SG)
tracking step by using a mini-batch of samples, where the batch size is
designed to be proportional to the size of the local dataset. We explicitly
evaluate the convergence rate of DSGT with respect to the number of iterations
in terms of algebraic connectivity of the network, mini-batch size, gradient
variance, etc. Under certain conditions, we further show that DSGT has a
network independence property in the sense that the network topology only
affects the convergence rate up to a constant factor. Hence, the convergence
rate of DSGT can be comparable to the centralized SGD method. Moreover, a
linear speedup of DSGT with respect to the number of nodes is achievable for
some scenarios. Numerical experiments for neural networks and logistic
regression problems on CIFAR-10 finally illustrate the advantages of DSGT.
