Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed
  Deep Learning
  Alpa automates model-parallel training of large deep learning (DL) models by
generating execution plans that unify data, operator, and pipeline parallelism.
Existing model-parallel training systems either require users to manually
create a parallelization plan or automatically generate one from a limited
space of model parallelism configurations. They do not suffice to scale out
complex DL models on distributed compute devices. Alpa distributes the training
of large DL models by viewing parallelisms as two hierarchical levels:
inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a
new hierarchical space for massive model-parallel execution plans. Alpa designs
a number of compilation passes to automatically derive efficient parallel
execution plans at each parallelism level. Alpa implements an efficient runtime
to orchestrate the two-level parallel execution on distributed compute devices.
Our evaluation shows Alpa generates parallelization plans that match or
outperform hand-tuned model-parallel training systems even on models they are
designed for. Unlike specialized systems, Alpa also generalizes to models with
heterogeneous architectures and models without manually-designed plans. Alpa's
source code is publicly available at https://github.com/alpa-projects/alpa
