Model-Parallel Model Selection for Deep Learning Systems
  As deep learning becomes more expensive, both in terms of time and compute,
inefficiencies in machine learning (ML) training prevent practical usage of
state-of-the-art models for most users. The newest model architectures are
simply too large to be fit onto a single processor. To address the issue, many
ML practitioners have turned to model parallelism as a method of distributing
the computational requirements across several devices. Unfortunately, the
sequential nature of neural networks causes very low efficiency and device
utilization in model parallel training jobs. We propose a new form of "shard
parallelism" combining task and model parallelism, then package it into a
framework we name Hydra. Hydra recasts the problem of model parallelism in the
multi-model context to produce a fine-grained parallel workload of independent
model shards, rather than independent models. This new parallel design promises
dramatic speedups relative to the traditional model parallelism paradigm.
