Effective Elastic Scaling of Deep Learning Workloads
  The increased use of deep learning (DL) in academia, government and industry
has, in turn, led to the popularity of on-premise and cloud-hosted deep
learning platforms, whose goals are to enable organizations utilize expensive
resources effectively, and to share said resources among multiple teams in a
fair and effective manner.
  In this paper, we examine the elastic scaling of Deep Learning (DL) jobs over
large-scale training platforms and propose a novel resource allocation strategy
for DL training jobs, resulting in improved job run time performance as well as
increased cluster utilization. We begin by analyzing DL workloads and exploit
the fact that DL jobs can be run with a range of batch sizes without affecting
their final accuracy. We formulate an optimization problem that explores a
dynamic batch size allocation to individual DL jobs based on their scaling
efficiency, when running on multiple nodes. We design a fast dynamic
programming based optimizer to solve this problem in real-time to determine
jobs that can be scaled up/down, and use this optimizer in an autoscaler to
dynamically change the allocated resources and batch sizes of individual DL
jobs.
  We demonstrate empirically that our elastic scaling algorithm can complete up
to $\approx 2 \times$ as many jobs as compared to a strong baseline algorithm
that also scales the number of GPUs but does not change the batch size. We also
demonstrate that the average completion time with our algorithm is up to
$\approx 10 \times$ faster than that of the baseline.
