FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks
  Transformers achieve remarkable performance in various domains, including
NLP, CV, audio processing, and graph analysis. However, they do not scale well
on long sequence tasks due to their quadratic complexity w.r.t. the inputs
length. Linear Transformers were proposed to address this limitation. However,
these models have shown weaker performance on the long sequence tasks comparing
to the original one. In this paper, we explore Linear Transformer models,
rethinking their two core components. Firstly, we improved Linear Transformer
with Shift-Invariant Kernel Function SIKF, which achieve higher accuracy
without loss in speed. Secondly, we introduce FastRPB which stands for Fast
Relative Positional Bias, which efficiently adds positional information to
self-attention using Fast Fourier Transformation. FastRPB is independent of the
self-attention mechanism and can be combined with an original self-attention
and all its efficient variants. FastRPB has O(N log(N)) computational
complexity, requiring O(N) memory w.r.t. input sequence length N.
