Addressing Algorithmic Bottlenecks in Elastic Machine Learning with
  Chicle
  Distributed machine learning training is one of the most common and important
workloads running on data centers today, but it is rarely executed alone.
Instead, to reduce costs, computing resources are consolidated and shared by
different applications. In this scenario, elasticity and proper load balancing
are vital to maximize efficiency, fairness, and utilization. Currently, most
distributed training frameworks do not support the aforementioned properties. A
few exceptions that do support elasticity, imitate generic distributed
frameworks and use micro-tasks. In this paper we illustrate that micro-tasks
are problematic for machine learning applications, because they require a high
degree of parallelism which hinders the convergence of distributed training at
a pure algorithmic level (i.e., ignoring overheads and scalability
limitations). To address this, we propose Chicle, a new elastic distributed
training framework which exploits the nature of machine learning algorithms to
implement elasticity and load balancing without micro-tasks. We use Chicle to
train deep neural network as well as generalized linear models, and show that
Chicle achieves performance competitive with state of the art rigid frameworks,
while efficiently enabling elastic execution and dynamic load balancing.
