Mesa: A Memory-saving Training Framework for Transformers
  There has been an explosion of interest in designing high-performance
Transformers. While Transformers have delivered significant performance
improvements, training such networks is extremely memory intensive owing to
storing all intermediate activations that are needed for gradient computation
during backpropagation, especially for long sequences. To this end, we present
Mesa, a memory-saving training framework for Transformers. Specifically, Mesa
uses exact activations during forward pass while storing a low-precision
version of activations to reduce memory consumption during training. The
low-precision activations are then dequantized during back-propagation to
compute gradients. Besides, to address the heterogeneous activation
distributions in the multi-head self-attention layers, we propose a head-wise
activation quantization strategy, which quantizes activations based on the
statistics of each head to minimize the approximation error. To further boost
training efficiency, we learn quantization parameters by running estimates.
More importantly, by re-investing the saved memory in employing a larger batch
size or scaling up model size, we may further improve the performance under
constrained computational resources. Extensive experiments on ImageNet,
CIFAR-100 and ADE20K demonstrate that Mesa can achieve flexible memory-savings
(up to 50%) during training while achieving comparable or even better
performance. Code is available at https://github.com/zip-group/Mesa.
