Exploring Multi-dimensional Hierarchical Network Topologies for
  Efficient Distributed Training of Trillion Parameter DL Models
  Deep Neural Networks have gained significant attraction due to their wide
applicability in different domains. DNN sizes and training samples are
constantly growing, making training of such workloads more challenging.
Distributed training is a solution to reduce the training time.
High-performance distributed training platforms should leverage
multi-dimensional hierarchical networks, which interconnect accelerators
through different levels of the network, to dramatically reduce expensive NICs
required for the scale-out network. However, it comes at the expense of
communication overhead between distributed accelerators to exchange gradients
or input/output activation. In order to allow for further scaling of the
workloads, communication overhead needs to be minimized. In this paper, we
motivate the fact that in training platforms, adding more intermediate network
dimensions is beneficial for efficiently mitigating the excessive use of
expensive NIC resources. Further, we address different challenges of the DNN
training on hierarchical networks. We discuss when designing the interconnect,
how to distribute network bandwidth resources across different dimensions in
order to (i) maximize BW utilization of all dimensions, and (ii) minimizing the
overall training time for the target workload. We then implement a framework
that, for a given workload, determines the best network configuration that
maximizes performance, or performance-per-cost.
