Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural
  Network Training
  Distributed deep neural network (DDNN) training constitutes an increasingly
important workload that frequently runs in the cloud. Larger DNN models and
faster compute engines are shifting DDNN training bottlenecks from computation
to communication. This paper characterizes DDNN training to precisely pinpoint
these bottlenecks. We found that timely training requires high performance
parameter servers (PSs) with optimized network stacks and gradient processing
pipelines, as well as server and network hardware with balanced computation and
communication resources. We therefore propose PHub, a high performance
multi-tenant, rack-scale PS design. PHub co-designs the PS software and
hardware to accelerate rack-level and hierarchical cross-rack parameter
exchange, with an API compatible with many DDNN training frameworks. PHub
provides a performance improvement of up to 2.7x compared to state-of-the-art
distributed training techniques for cloud-based ImageNet workloads, with 25%
better throughput per dollar.
