Revitalizing CNN Attentions via Transformers in Self-Supervised Visual
  Representation Learning
  Studies on self-supervised visual representation learning (SSL) improve
encoder backbones to discriminate training samples without labels. While CNN
encoders via SSL achieve comparable recognition performance to those via
supervised learning, their network attention is under-explored for further
improvement. Motivated by the transformers that explore visual attention
effectively in recognition scenarios, we propose a CNN Attention REvitalization
(CARE) framework to train attentive CNN encoders guided by transformers in SSL.
The proposed CARE framework consists of a CNN stream (C-stream) and a
transformer stream (T-stream), where each stream contains two branches.
C-stream follows an existing SSL framework with two CNN encoders, two
projectors, and a predictor. T-stream contains two transformers, two
projectors, and a predictor. T-stream connects to CNN encoders and is in
parallel to the remaining C-Stream. During training, we perform SSL in both
streams simultaneously and use the T-stream output to supervise C-stream. The
features from CNN encoders are modulated in T-stream for visual attention
enhancement and become suitable for the SSL scenario. We use these modulated
features to supervise C-stream for learning attentive CNN encoders. To this
end, we revitalize CNN attention by using transformers as guidance. Experiments
on several standard visual recognition benchmarks, including image
classification, object detection, and semantic segmentation, show that the
proposed CARE framework improves CNN encoder backbones to the state-of-the-art
performance.
