CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale
  Attention
  Transformers have made great progress in dealing with computer vision tasks.
However, existing vision transformers do not yet possess the ability of
building the interactions among features of different scales, which is
perceptually important to visual inputs. The reasons are two-fold: (1) Input
embeddings of each layer are equal-scale, so no cross-scale feature can be
extracted; (2) to lower the computational cost, some vision transformers merge
adjacent embeddings inside the self-attention module, thus sacrificing
small-scale (fine-grained) features of the embeddings and also disabling the
cross-scale interactions. To this end, we propose Cross-scale Embedding Layer
(CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends
each embedding with multiple patches of different scales, providing the
self-attention module itself with cross-scale features. On the other hand, LSDA
splits the self-attention module into a short-distance one and a long-distance
counterpart, which not only reduces the computational burden but also keeps
both small-scale and large-scale features in the embeddings. Through the above
two designs, we achieve cross-scale attention. Besides, we put forward a
dynamic position bias for vision transformers to make the popular relative
position bias apply to variable-sized images. Hinging on the cross-scale
attention module, we construct a versatile vision architecture, dubbed
CrossFormer, which accommodates variable-sized inputs. Extensive experiments
show that CrossFormer outperforms the other vision transformers on image
classification, object detection, instance segmentation, and semantic
segmentation tasks. The code has been released:
https://github.com/cheerss/CrossFormer.
