Crossover-SGD: A gossip-based communication in distributed deep learning
  for alleviating large mini-batch problem and enhancing scalability
  Distributed deep learning is an effective way to reduce the training time of
deep learning for large datasets as well as complex models. However, the
limited scalability caused by network overheads makes it difficult to
synchronize the parameters of all workers. To resolve this problem,
gossip-based methods that demonstrates stable scalability regardless of the
number of workers have been proposed. However, to use gossip-based methods in
general cases, the validation accuracy for a large mini-batch needs to be
verified. To verify this, we first empirically study the characteristics of
gossip methods in a large mini-batch problem and observe that the gossip
methods preserve higher validation accuracy than AllReduce-SGD(Stochastic
Gradient Descent) when the number of batch sizes is increased and the number of
workers is fixed. However, the delayed parameter propagation of the
gossip-based models decreases validation accuracy in large node scales. To cope
with this problem, we propose Crossover-SGD that alleviates the delay
propagation of weight parameters via segment-wise communication and load
balancing random network topology. We also adapt hierarchical communication to
limit the number of workers in gossip-based communication methods. To validate
the effectiveness of our proposed method, we conduct empirical experiments and
observe that our Crossover-SGD shows higher node scalability than
SGP(Stochastic Gradient Push).
