Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning
  Preprocessing Pipelines
  Preprocessing pipelines in deep learning aim to provide sufficient data
throughput to keep the training processes busy. Maximizing resource utilization
is becoming more challenging as the throughput of training processes increases
with hardware innovations (e.g., faster GPUs, TPUs, and inter-connects) and
advanced parallelization techniques that yield better scalability. At the same
time, the amount of training data needed in order to train increasingly complex
models is growing. As a consequence of this development, data preprocessing and
provisioning are becoming a severe bottleneck in end-to-end deep learning
pipelines.
  In this paper, we provide an in-depth analysis of data preprocessing
pipelines from four different machine learning domains. We introduce a new
perspective on efficiently preparing datasets for end-to-end deep learning
pipelines and extract individual trade-offs to optimize throughput,
preprocessing time, and storage consumption. Additionally, we provide an
open-source profiling library that can automatically decide on a suitable
preprocessing strategy to maximize throughput. By applying our generated
insights to real-world use-cases, we obtain an increased throughput of 3x to
13x compared to an untuned system while keeping the pipeline functionally
identical. These findings show the enormous potential of data pipeline tuning.
