Efficient Self-supervised Vision Transformers for Representation
  Learning
  This paper investigates two techniques for developing efficient
self-supervised vision transformers (EsViT) for visual representation learning.
First, we show through a comprehensive empirical study that multi-stage
architectures with sparse self-attentions can significantly reduce modeling
complexity but with a cost of losing the ability to capture fine-grained
correspondences between image regions. Second, we propose a new pre-training
task of region matching which allows the model to capture fine-grained region
dependencies and as a result significantly improves the quality of the learned
vision representations. Our results show that combining the two techniques,
EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation,
outperforming prior arts with around an order magnitude of higher throughput.
When transferring to downstream linear classification tasks, EsViT outperforms
its supervised counterpart on 17 out of 18 datasets. The code and models will
be publicly available.
