XCiT: Cross-Covariance Image Transformers
  Following their success in natural language processing, transformers have
recently shown much promise for computer vision. The self-attention operation
underlying transformers yields global interactions between all tokens ,i.e.
words or image patches, and enables flexible modelling of image data beyond the
local interactions of convolutions. This flexibility, however, comes with a
quadratic complexity in time and memory, hindering application to long
sequences and high-resolution images. We propose a "transposed" version of
self-attention that operates across feature channels rather than tokens, where
the interactions are based on the cross-covariance matrix between keys and
queries. The resulting cross-covariance attention (XCA) has linear complexity
in the number of tokens, and allows efficient processing of high-resolution
images. Our cross-covariance image transformer (XCiT) is built upon XCA. It
combines the accuracy of conventional transformers with the scalability of
convolutional architectures. We validate the effectiveness and generality of
XCiT by reporting excellent results on multiple vision benchmarks, including
image classification and self-supervised feature learning on ImageNet-1k,
object detection and instance segmentation on COCO, and semantic segmentation
on ADE20k.
