Analyzing and Mitigating Data Stalls in DNN Training
  Training Deep Neural Networks (DNNs) is resource-intensive and
time-consuming. While prior research has explored many different ways of
reducing DNN training time, the impact of input data pipeline, i.e., fetching
raw data items from storage and performing data pre-processing in memory, has
been relatively unexplored. This paper makes the following contributions: (1)
We present the first comprehensive analysis of how the input data pipeline
affects the training time of widely-used computer vision and audio Deep Neural
Networks (DNNs), that typically involve complex data preprocessing. We analyze
nine different models across three tasks and four datasets while varying
factors such as the amount of memory, number of CPU threads, storage device,
GPU generation etc on servers that are a part of a large production cluster at
Microsoft. We find that in many cases, DNN training time is dominated by data
stall time: time spent waiting for data to be fetched and preprocessed. (2) We
build a tool, DS-Analyzer to precisely measure data stalls using a differential
technique, and perform predictive what-if analysis on data stalls. (3) Finally,
based on the insights from our analysis, we design and implement three simple
but effective techniques in a data-loading library, CoorDL, to mitigate data
stalls. Our experiments on a range of DNN tasks, models, datasets, and hardware
configs show that when PyTorch uses CoorDL instead of the state-of-the-art DALI
data loading library, DNN training time is reduced significantly (by as much as
5x on a single server).
