Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN
  Training
  The Long-Short-Term-Memory Recurrent Neural Networks (LSTM RNNs) are a
popular class of machine learning models for analyzing sequential data. Their
training on modern GPUs, however, is limited by the GPU memory capacity. Our
profiling results of the LSTM RNN-based Neural Machine Translation (NMT) model
reveal that feature maps of the attention and RNN layers form the memory
bottleneck and runtime is unevenly distributed across different layers when
training on GPUs. Based on these two observations, we propose to recompute the
feature maps rather than stashing them persistently in the GPU memory.
  While the idea of feature map recomputation has been considered before,
existing solutions fail to deliver satisfactory footprint reduction, as they do
not address two key challenges. For each feature map recomputation to be
effective and efficient, its effect on (1) the total memory footprint, and (2)
the total execution time has to be carefully estimated. To this end, we propose
*Echo*, a new compiler-based optimization scheme that addresses the first
challenge with a practical mechanism that estimates the memory benefits of
recomputation over the entire computation graph, and the second challenge by
non-conservatively estimating the recomputation overhead leveraging layer
specifics. *Echo* reduces the GPU memory footprint automatically and
transparently without any changes required to the training source code, and is
effective for models beyond LSTM RNNs.
  We evaluate *Echo* on numerous state-of-the-art machine learning workloads on
real systems with modern GPUs and observe footprint reduction ratios of 1.89X
on average and 3.13X maximum. Such reduction can be converted into faster
training with a larger batch size, savings in GPU energy consumption (e.g.,
training with one GPU as fast as with four), and/or an increase in the maximum
number of layers under the same GPU memory budget.
