Augmented Shortcuts for Vision Transformers
  Transformer models have achieved great progress on computer vision tasks
recently. The rapid development of vision transformers is mainly contributed by
their high representation ability for extracting informative features from
input images. However, the mainstream transformer models are designed with deep
architectures, and the feature diversity will be continuously reduced as the
depth increases, i.e., feature collapse. In this paper, we theoretically
analyze the feature collapse phenomenon and study the relationship between
shortcuts and feature diversity in these transformer models. Then, we present
an augmented shortcut scheme, which inserts additional paths with learnable
parameters in parallel on the original shortcuts. To save the computational
costs, we further explore an efficient approach that uses the block-circulant
projection to implement augmented shortcuts. Extensive experiments conducted on
benchmark datasets demonstrate the effectiveness of the proposed method, which
brings about 1% accuracy increase of the state-of-the-art visual transformers
without obviously increasing their parameters and FLOPs.
