Revisiting Linformer with a modified self-attention with linear
  complexity
  Although Transformer models such as Google's BERT and OpenAI's GPT-3 are
successful in many natural language processing tasks, training and deploying
these models are costly and inefficient.Even if pre-trained models are used,
deploying these models still remained a challenge due to their large size.
Apart from deployment, these models take higher time during inference
restricting user-friendliness. The main bottleneck is self-attention which uses
quadratic time and space with respect to the sequence length. In order to
reduce the quadratic time complexity of the self-attention mechanism, Linformer
by Facebook's AI research team was introduced where they showed that the
self-attention mechanism can be approximated by a low-rank matrix and
exploiting this finding, a new method for self-attention with linear time and
space complexity was proposed by them. In the Linformer, the time complexity
depends on the projection mapping dimension which acts as a hyperparameter and
affects the performance of the model, tuning this hyperparameter can be
time-consuming. In this paper, I proposed an alternative method for
self-attention with linear complexity in time and space and is independent of
the projection mapping dimension. Since this method works for long sequences
this can be used for images as well as audios.
