Staircase Attention for Recurrent Processing of Sequences
  Attention mechanisms have become a standard tool for sequence modeling tasks,
in particular by stacking self-attention layers over the entire input sequence
as in the Transformer architecture. In this work we introduce a novel attention
procedure called staircase attention that, unlike self-attention, operates
across the sequence (in time) recurrently processing the input by adding
another step of processing. A step in the staircase comprises of backward
tokens (encoding the sequence so far seen) and forward tokens (ingesting a new
part of the sequence), or an extreme Ladder version with a forward step of zero
that simply repeats the Transformer on each step of the ladder, sharing the
weights. We thus describe a family of such models that can trade off
performance and compute, by either increasing the amount of recurrence through
time, the amount of sequential processing via recurrence in depth, or both.
Staircase attention is shown to be able to solve tasks that involve tracking
that conventional Transformers cannot, due to this recurrence. Further, it is
shown to provide improved modeling power for the same size model (number of
parameters) compared to self-attentive Transformers on large language modeling
and dialogue tasks, yielding significant perplexity gains.
