Importance of Data Loading Pipeline in Training Deep Neural Networks
  Training large-scale deep neural networks is a long, time-consuming
operation, often requiring many GPUs to accelerate. In large models, the time
spent loading data takes a significant portion of model training time. As GPU
servers are typically expensive, tricks that can save training time are
valuable.Slow training is observed especially on real-world applications where
exhaustive data augmentation operations are required. Data augmentation
techniques include: padding, rotation, adding noise, down sampling, up
sampling, etc. These additional operations increase the need to build an
efficient data loading pipeline, and to explore existing tools to speed up
training time. We focus on the comparison of two main tools designed for this
task, namely binary data format to accelerate data reading, and NVIDIA DALI to
accelerate data augmentation. Our study shows improvement on the order of 20%
to 40% if such dedicated tools are used.
