Benchmarking network fabrics for data distributed training of deep
  neural networks
  Artificial Intelligence/Machine Learning applications require the training of
complex models on large amounts of labelled data. The large computational
requirements for training deep models have necessitated the development of new
methods for faster training. One such approach is the data parallel approach,
where the training data is distributed across multiple compute nodes. This
approach is simple to implement and supported by most of the commonly used
machine learning frameworks. The data parallel approach leverages MPI for
communicating gradients across all nodes. In this paper, we examine the effects
of using different physical hardware interconnects and network-related software
primitives for enabling data distributed deep learning. We compare the effect
of using GPUDirect and NCCL on Ethernet and OmniPath fabrics. Our results show
that using Ethernet-based networking in shared HPC systems does not have a
significant effect on the training times for commonly used deep neural network
architectures or traditional HPC applications such as Computational Fluid
Dynamics.
