An Empirical Study of Training Self-Supervised Vision Transformers
  This paper does not describe a novel method. Instead, it studies a
straightforward, incremental, yet must-know baseline given the recent progress
in computer vision: self-supervised learning for Vision Transformers (ViT).
While the training recipes for standard convolutional networks have been highly
mature and robust, the recipes for ViT are yet to be built, especially in the
self-supervised scenarios where training becomes more challenging. In this
work, we go back to basics and investigate the effects of several fundamental
components for training self-supervised ViT. We observe that instability is a
major issue that degrades accuracy, and it can be hidden by apparently good
results. We reveal that these results are indeed partial failure, and they can
be improved when training is made more stable. We benchmark ViT results in MoCo
v3 and several other self-supervised frameworks, with ablations in various
aspects. We discuss the currently positive evidence as well as challenges and
open questions. We hope that this work will provide useful data points and
experience for future research.
