VirtualFlow: Decoupling Deep Learning Models from the Underlying
  Hardware
  State-of-the-art deep learning systems such as TensorFlow and PyTorch tightly
couple the model with the underlying hardware. This coupling requires the user
to modify application logic in order to run the same job across a different set
of resources, thereby limiting the choice of hardware for a given workload and
potentially forcing the user to forgo more efficient hardware configurations.
  We propose VirtualFlow, a system leveraging a novel abstraction called
virtual node processing to decouple the model from the hardware. In each step
of training or inference, the batch of input data is split across virtual nodes
instead of hardware accelerators (e.g. GPUs and TPUs). Mapping multiple virtual
nodes to each accelerator and processing them sequentially effectively time
slices the batch, thereby allowing users to reduce the memory requirement of
their workloads and mimic large batch sizes on small clusters.
  Using this technique, VirtualFlow enables many new use cases, such as
reproducing training results across different hardware, resource elasticity,
and heterogeneous training. In our evaluation, our implementation of
VirtualFlow for TensorFlow achieved strong convergence guarantees across
different hardware with out-of-the-box hyperparameters, up to 48% lower job
completion times with resource elasticity, and up to 42% higher throughput with
heterogeneous training.
