Hoplite: Efficient and Fault-Tolerant Collective Communication for
  Task-Based Distributed Systems
  Task-based distributed frameworks (e.g., Ray, Dask, Hydro) have become
increasingly popular for distributed applications that contain asynchronous and
dynamic workloads, including asynchronous gradient descent, reinforcement
learning, and model serving. As more data-intensive applications move to run on
top of task-based systems, collective communication efficiency has become an
important problem. Unfortunately, traditional collective communication
libraries (e.g., MPI, Horovod, NCCL) are an ill fit, because they require the
communication schedule to be known before runtime and they do not provide fault
tolerance.
  We design and implement Hoplite, an efficient and fault-tolerant collective
communication layer for task-based distributed systems. Our key technique is to
compute data transfer schedules on the fly and execute the schedules
efficiently through fine-grained pipelining. At the same time, when a task
fails, the data transfer schedule adapts quickly to allow other tasks to keep
making progress. We apply Hoplite to a popular task-based distributed
framework, Ray. We show that Hoplite speeds up asynchronous stochastic gradient
descent, reinforcement learning, and serving an ensemble of machine learning
models that are difficult to execute efficiently with traditional collective
communication by up to 7.8x, 3.9x, and 3.3x, respectively.
