PSO-PS: Parameter Synchronization with Particle Swarm Optimization for
  Distributed Training of Deep Neural Networks
  Parameter updating is an important stage in parallelism-based distributed
deep learning. Synchronous methods are widely used in distributed training the
Deep Neural Networks (DNNs). To reduce the communication and synchronization
overhead of synchronous methods, decreasing the synchronization frequency
(e.g., every $n$ mini-batches) is a straightforward approach. However, it often
suffers from poor convergence. In this paper, we propose a new algorithm of
integrating Particle Swarm Optimization (PSO) into the distributed training
process of DNNs to automatically compute new parameters. In the proposed
algorithm, a computing work is encoded by a particle, the weights of DNNs and
the training loss are modeled by the particle attributes. At each
synchronization stage, the weights are updated by PSO from the sub weights
gathered from all workers, instead of averaging the weights or the gradients.
To verify the performance of the proposed algorithm, the experiments are
performed on two commonly used image classification benchmarks: MNIST and
CIFAR10, and compared with the peer competitors at multiple different
synchronization configurations. The experimental results demonstrate the
competitiveness of the proposed algorithm.
