Unleashing the Power of Contrastive Self-Supervised Visual Models via
  Contrast-Regularized Fine-Tuning
  Contrastive self-supervised learning (CSL) has attracted increasing attention
for model pre-training via unlabeled data. The resulted CSL models provide
instance-discriminative visual features that are uniformly scattered in the
feature space. During deployment, the common practice is to directly fine-tune
CSL models with cross-entropy, which however may not be the best strategy in
practice. Although cross-entropy tends to separate inter-class features, the
resulting models still have limited capability for reducing intra-class feature
scattering that exists in CSL models. In this paper, we investigate whether
applying contrastive learning to fine-tuning would bring further benefits, and
analytically find that optimizing the contrastive loss benefits both
discriminative representation learning and model optimization during
fine-tuning. Inspired by these findings, we propose Contrast-regularized tuning
(Core-tuning), a new approach for fine-tuning CSL models. Instead of simply
adding the contrastive loss to the objective of fine-tuning, Core-tuning
further applies a novel hard pair mining strategy for more effective
contrastive fine-tuning, as well as smoothing the decision boundary to better
exploit the learned discriminative feature space. Extensive experiments on
image classification and semantic segmentation verify the effectiveness of
Core-tuning.
