Weighted parallel SGD for distributed unbalanced-workload training
  system
  Stochastic gradient descent (SGD) is a popular stochastic optimization method
in machine learning. Traditional parallel SGD algorithms, e.g., SimuParallel
SGD, often require all nodes to have the same performance or to consume equal
quantities of data. However, these requirements are difficult to satisfy when
the parallel SGD algorithms run in a heterogeneous computing environment;
low-performance nodes will exert a negative influence on the final result. In
this paper, we propose an algorithm called weighted parallel SGD (WP-SGD).
WP-SGD combines weighted model parameters from different nodes in the system to
produce the final output. WP-SGD makes use of the reduction in standard
deviation to compensate for the loss from the inconsistency in performance of
nodes in the cluster, which means that WP-SGD does not require that all nodes
consume equal quantities of data. We also analyze the theoretical feasibility
of running two other parallel SGD algorithms combined with WP-SGD in a
heterogeneous environment. The experimental results show that WP-SGD
significantly outperforms the traditional parallel SGD algorithms on
distributed training systems with an unbalanced workload.
