Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI
  Collective Performance during Tensor Accumulation for Parallelized Training
  of Neural Machine Translation Models
  Neural machine translation - using neural networks to translate human
language - is an area of active research exploring new neuron types and network
topologies with the goal of dramatically improving machine translation
performance. Current state-of-the-art approaches, such as the multi-head
attention-based transformer, require very large translation corpuses and many
epochs to produce models of reasonable quality. Recent attempts to parallelize
the official TensorFlow "Transformer" model across multiple nodes have hit
roadblocks due to excessive memory use and resulting out of memory errors when
performing MPI collectives. This paper describes modifications made to the
Horovod MPI-based distributed training framework to reduce memory usage for
transformer models by converting assumed-sparse tensors to dense tensors, and
subsequently replacing sparse gradient gather with dense gradient reduction.
The result is a dramatic increase in scale-out capability, with CPU-only
scaling tests achieving 91% weak scaling efficiency up to 1200 MPI processes
(300 nodes), and up to 65% strong scaling efficiency up to 400 MPI processes
(200 nodes) using the Stampede2 supercomputer.
