Unifying Data, Model and Hybrid Parallelism in Deep Learning via Tensor
  Tiling
  Deep learning systems have become vital tools across many fields, but the
increasing model sizes mean that training must be accelerated to maintain such
systems' utility. Current systems like Tensorflow and MXNet focus on one
specific parallelization strategy, data parallelism, which requires large
training batch sizes in order to scale. We cast the problem of finding the best
parallelization strategy as the problem of finding the best tiling to partition
tensors with the least overall communication. We propose an algorithm that can
find the optimal tiling. Our resulting parallelization solution is a hybrid of
data parallelism and model parallelism. We build the SoyBean system that
performs automatic parallelization. SoyBean automatically transforms a serial
dataflow graph captured by an existing deep learning system frontend into a
parallel dataflow graph based on the optimal tiling it has found. Our
evaluations show that SoyBean is 1.5x-4x faster than pure data parallelism for
AlexNet and VGG. We present this automatic tiling in a new system, SoyBean,
that can act as a backend for Tensorflow, MXNet, and others.
