Sliced Recursive Transformer
  We present a neat yet effective recursive operation on vision transformers
that can improve parameter utilization without involving additional parameters.
This is achieved by sharing weights across depth of transformer networks. The
proposed method can obtain a substantial gain (~2%) simply using na\"ive
recursive operation, requires no special or sophisticated knowledge for
designing principles of networks, and introduces minimum computational overhead
to the training procedure. To reduce the additional computation caused by
recursive operation while maintaining the superior accuracy, we propose an
approximating method through multiple sliced group self-attentions across
recursive layers which can reduce the cost consumption by 10~30% with minimal
performance loss. We call our model Sliced Recursive Transformer (SReT), which
is compatible with a broad range of other designs for efficient vision
transformers. Our best model establishes significant improvement on ImageNet
over state-of-the-art methods while containing fewer parameters. The proposed
sliced recursive operation allows us to build a transformer with more than 100
or even 1000 layers effortlessly under a still small size (13~15M), to avoid
difficulties in optimization when the model size is too large. The flexible
scalability has shown great potential for scaling up and constructing extremely
deep and large dimensionality vision transformers. Our code and models are
available at https://github.com/szq0214/SReT.
