Shatter: An Efficient Transformer Encoder with Single-Headed
  Self-Attention and Relative Sequence Partitioning
  The highly popular Transformer architecture, based on self-attention, is the
foundation of large pretrained models such as BERT, that have become an
enduring paradigm in NLP. While powerful, the computational resources and time
required to pretrain such models can be prohibitive. In this work, we present
an alternative self-attention architecture, Shatter, that more efficiently
encodes sequence information by softly partitioning the space of relative
positions and applying different value matrices to different parts of the
sequence. This mechanism further allows us to simplify the multi-headed
attention in Transformer to single-headed. We conduct extensive experiments
showing that Shatter achieves better performance than BERT, with pretraining
being faster per step (15% on TPU), converging in fewer steps, and offering
considerable memory savings (>50%). Put together, Shatter can be pretrained on
8 V100 GPUs in 7 days, and match the performance of BERT_Base -- making the
cost of pretraining much more affordable.
