Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via
  Feature Distillation
  Masked image modeling (MIM) learns representations with remarkably good
fine-tuning performances, overshadowing previous prevalent pre-training
approaches such as image classification, instance contrastive learning, and
image-text alignment. In this paper, we show that the inferior fine-tuning
performance of these pre-training approaches can be significantly improved by a
simple post-processing in the form of feature distillation (FD). The feature
distillation converts the old representations to new representations that have
a few desirable properties just like those representations produced by MIM.
These properties, which we aggregately refer to as optimization friendliness,
are identified and analyzed by a set of attention- and optimization-related
diagnosis tools. With these properties, the new representations show strong
fine-tuning performance. Specifically, the contrastive self-supervised learning
methods are made as competitive in fine-tuning as the state-of-the-art masked
image modeling (MIM) algorithms. The CLIP models' fine-tuning performance is
also significantly improved, with a CLIP ViT-L model reaching 89.0% top-1
accuracy on ImageNet-1K classification. More importantly, our work provides a
way for the future research to focus more effort on the generality and
scalability of the learnt representations without being pre-occupied with
optimization friendliness since it can be enhanced rather easily. The code will
be available at https://github.com/SwinTransformer/Feature-Distillation.
