Adaptive Fourier Neural Operators: Efficient Token Mixers for
  Transformers
  Vision transformers have delivered tremendous success in representation
learning. This is primarily due to effective token mixing through self
attention. However, this scales quadratically with the number of pixels, which
becomes infeasible for high-resolution inputs. To cope with this challenge, we
propose Adaptive Fourier Neural Operator (AFNO) as an efficient token mixer
that learns to mix in the Fourier domain. AFNO is based on a principled
foundation of operator learning which allows us to frame token mixing as a
continuous global convolution without any dependence on the input resolution.
This principle was previously used to design FNO, which solves global
convolution efficiently in the Fourier domain and has shown promise in learning
challenging PDEs. To handle challenges in visual representation learning such
as discontinuities in images and high resolution inputs, we propose principled
architectural modifications to FNO which results in memory and computational
efficiency. This includes imposing a block-diagonal structure on the channel
mixing weights, adaptively sharing weights across tokens, and sparsifying the
frequency modes via soft-thresholding and shrinkage. The resulting model is
highly parallel with a quasi-linear complexity and has linear memory in the
sequence size. AFNO outperforms self-attention mechanisms for few-shot
segmentation in terms of both efficiency and accuracy. For Cityscapes
segmentation with the Segformer-B3 backbone, AFNO can handle a sequence size of
65k and outperforms other efficient self-attention mechanisms.
