Speeding Up Distributed Gradient Descent by Utilizing Non-persistent
  Stragglers
  Distributed gradient descent (DGD) is an efficient way of implementing
gradient descent (GD), especially for large data sets, by dividing the
computation tasks into smaller subtasks and assigning to different computing
servers (CSs) to be executed in parallel. In standard parallel execution,
per-iteration waiting time is limited by the execution time of the straggling
servers. Coded DGD techniques have been introduced recently, which can tolerate
straggling servers via assigning redundant computation tasks to the CSs. In
most of the existing DGD schemes, either with coded computation or coded
communication, the non-straggling CSs transmit one message per iteration once
they complete all their assigned computation tasks. However, although the
straggling servers cannot complete all their assigned tasks, they are often
able to complete a certain portion of them. In this paper, we allow multiple
transmissions from each CS at each iteration in order to make sure a maximum
number of completed computations can be reported to the aggregating server
(AS), including the straggling servers. We numerically show that the average
completion time per iteration can be reduced significantly by slightly
increasing the communication load per server.
