Dynamic backup workers for parallel machine learning
  The most popular framework for distributed training of machine learning
models is the (synchronous) parameter server (PS). This paradigm consists of
$n$ workers, which iteratively compute updates of the model parameters, and a
stateful PS, which waits and aggregates all updates to generate a new estimate
of model parameters and sends it back to the workers for a new iteration.
Transient computation slowdowns or transmission delays can intolerably lengthen
the time of each iteration. An efficient way to mitigate this problem is to let
the PS wait only for the fastest $n-b$ updates, before generating the new
parameters. The slowest $b$ workers are called backup workers. The optimal
number $b$ of backup workers depends on the cluster configuration and workload,
but also (as we show in this paper) on the hyper-parameters of the learning
algorithm and the current stage of the training. We propose DBW, an algorithm
that dynamically decides the number of backup workers during the training
process to maximize the convergence speed at each iteration. Our experiments
show that DBW 1) removes the necessity to tune $b$ by preliminary
time-consuming experiments, and 2) makes the training up to a factor $3$ faster
than the optimal static configuration.
