Automatic Model Parallelism for Deep Neural Networks with Compiler and
  Hardware Support
  The deep neural networks (DNNs) have been enormously successful in tasks that
were hitherto in the human-only realm such as image recognition, and language
translation. Owing to their success the DNNs are being explored for use in ever
more sophisticated tasks. One of the ways that the DNNs are made to scale for
the complex undertakings is by increasing their size -- deeper and wider
networks can model well the additional complexity. Such large models are
trained using model parallelism on multiple compute devices such as multi-GPUs
and multi-node systems.
  In this paper, we develop a compiler-driven approach to achieve model
parallelism. We model the computation and communication costs of a dataflow
graph that embodies the neural network training process and then, partition the
graph using heuristics in such a manner that the communication between compute
devices is minimal and we have a good load balance. The hardware scheduling
assistants are proposed to assist the compiler in fine tuning the distribution
of work at runtime.
