Characterizing and Modeling Distributed Training with Transient Cloud
  GPU Servers
  Cloud GPU servers have become the de facto way for deep learning
practitioners to train complex models on large-scale datasets. However, it is
challenging to determine the appropriate cluster configuration---e.g., server
type and number---for different training workloads while balancing the
trade-offs in training time, cost, and model accuracy. Adding to the complexity
is the potential to reduce the monetary cost by using cheaper, but revocable,
transient GPU servers.
  In this work, we analyze distributed training performance under diverse
cluster configurations using CM-DARE, a cloud-based measurement and training
framework. Our empirical datasets include measurements from three GPU types,
six geographic regions, twenty convolutional neural networks, and thousands of
Google Cloud servers. We also demonstrate the feasibility of predicting
training speed and overhead using regression-based models. Finally, we discuss
potential use cases of our performance modeling such as detecting and
mitigating performance bottlenecks.
