Agglomerative Attention
  Neural networks using transformer-based architectures have recently
demonstrated great power and flexibility in modeling sequences of many types.
One of the core components of transformer networks is the attention layer,
which allows contextual information to be exchanged among sequence elements.
While many of the prevalent network structures thus far have utilized full
attention -- which operates on all pairs of sequence elements -- the quadratic
scaling of this attention mechanism significantly constrains the size of models
that can be trained. In this work, we present an attention model that has only
linear requirements in memory and computation time. We show that, despite the
simpler attention model, networks using this attention mechanism can attain
comparable performance to full attention networks on language modeling tasks.
