OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
  Deep learning frameworks such as TensorFlow and PyTorch provide a productive
interface for expressing and training a deep neural network (DNN) model on a
single device or using data parallelism. Still, they may not be flexible or
efficient enough in training emerging large models on distributed devices,
which require more sophisticated parallelism beyond data parallelism. Plugins
or wrappers have been developed to strengthen these frameworks for model or
pipeline parallelism, but they complicate the usage and implementation of
distributed deep learning. Aiming at a simple, neat redesign of distributed
deep learning frameworks for various parallelism paradigms, we present OneFlow,
a novel distributed training framework based on an SBP (split, broadcast and
partial-value) abstraction and the actor model. SBP enables much easier
programming of data parallelism and model parallelism than existing frameworks,
and the actor model provides a succinct runtime mechanism to manage the complex
dependencies imposed by resource constraints, data movement and computation in
distributed deep learning. We demonstrate the general applicability and
efficiency of OneFlow for training various large DNN models with case studies
and extensive experiments. The results show that OneFlow outperforms many
well-known customized libraries built on top of the state-of-the-art
frameworks. The code of OneFlow is available at:
https://github.com/Oneflow-Inc/oneflow.
