RPC Considered Harmful: Fast Distributed Deep Learning on RDMA
  Deep learning emerges as an important new resource-intensive workload and has
been successfully applied in computer vision, speech, natural language
processing, and so on. Distributed deep learning is becoming a necessity to
cope with growing data and model sizes. Its computation is typically
characterized by a simple tensor data abstraction to model multi-dimensional
matrices, a data-flow graph to model computation, and iterative executions with
relatively frequent synchronizations, thereby making it substantially different
from Map/Reduce style distributed big data computation.
  RPC, commonly used as the communication primitive, has been adopted by
popular deep learning frameworks such as TensorFlow, which uses gRPC. We show
that RPC is sub-optimal for distributed deep learning computation, especially
on an RDMA-capable network. The tensor abstraction and data-flow graph, coupled
with an RDMA network, offers the opportunity to reduce the unnecessary overhead
(e.g., memory copy) without sacrificing programmability and generality. In
particular, from a data access point of view, a remote machine is abstracted
just as a "device" on an RDMA channel, with a simple memory interface for
allocating, reading, and writing memory regions. Our graph analyzer looks at
both the data flow graph and the tensors to optimize memory allocation and
remote data access using this interface. The result is up to 25 times speedup
in representative deep learning benchmarks against the standard gRPC in
TensorFlow and up to 169% improvement even against an RPC implementation
optimized for RDMA, leading to faster convergence in the training process.
