{'fraction_non_alphanumeric': 0.03651685393258427, 'fraction_numerical': 0.015122364168077574, 'mean_word_length': 4.9662534435261705, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Revitalizing CNN Attentions via Transformers in Self-Supervised Visual Representation Learning

Studies on self-supervised visual representation learning (SSL) improve encoder backbones to discriminate training samples without labels. While CNN encoders via SSL achieve comparable recognition performance to those via supervised learning, their network attention is under-explored for further improvement. Motivated by the transformers that explore visual attention effectively in recognition scenarios, we propose a CNN Attention REvitalization (CARE) framework to train attentive CNN encoders guided by transformers in SSL. The proposed CARE framework consists of a CNN stream (C-stream) and a transformer stream (T-stream), where each stream contains two branches. C-stream follows an existing SSL framework with two CNN encoders, two projectors, and a predictor. T-stream contains two transformers, two projectors, and a predictor. T-stream connects to CNN encoders and is in parallel to the remaining C-Stream. During training, we perform SSL in both streams simultaneously and use the T-stream output to supervise C-stream. The features from CNN encoders are modulated in T-stream for visual attention enhancement and become suitable for the SSL scenario. We use these modulated features to supervise C-stream for learning attentive CNN encoders. To this end, we revitalize CNN attention by using transformers as guidance. Experiments on several standard visual recognition benchmarks, including image classification, object detection, and semantic segmentation, show that the proposed CARE framework improves CNN encoder backbones to the state-of-the-art performance.


Introduction
Learning visual features effectively has a profound influence on the recognition performance [5,53]. Upon handling large-scale natural images, self-supervised visual representation learning benefits downstream recognition tasks via pretext feature training. Existing SSL methods typically leverage two branches to measure the similarity between different view representations derived from the same input image. By maximizing the similarity between the correlated views within one image (e.g., BYOL [24], SimSiam [13] and Barlow Twins [73]), or minimizing the similarity between views from different images (e.g., MoCo [29] and SimCLR [12]), these methods are shown to be effective towards learning self-supervised visual representations. SSL evolves concurrently with the transformer. Debuting at natural language processing [62,17], transformers have shown their advantages to process large-scale visual data since ViT [19]. The encoder-decoder architecture in this vision transformer consistently explores global attention without convolution. This architecture is shown effective for visual recognition with [7,76,55] or without CNN integration [38,21]. Inspired by these achievements via supervised learning, studies [15, 10, 4, [15] explores visual attention by explicitly taking a vision transformer as encoder, while SimCLR [12] and BYOL [24] do not learn an attentive CNN encoder. Our CARE framework consists of a C-stream and a T-stream to explore visual attention in CNN encoders with transformer supervision. Note that only target CNN encoder (i.e., CNN 1 ) is preserved after pre-training for downstream evaluation. We do not show projectors in (b) and (d) for simplicity. 71] arise recently to train transformers in a self-supervised manner. These methods maintain most of the SSL pipeline (i.e., encoder, projector, and predictor) utilized for training CNN encoders. Without significant framework alteration, original SSL methods for CNN encoders can be adapted to train transformer encoders and achieve favorable performance.
The success of using transformer encoders indicates that visual attention benefits encoder backbones in SSL. On the other hand, in supervised learning, CNN attention is usually developed via network supervision [47]. However, we observe that existing SSL methods do not incorporate visual attention within CNN encoders. This motivates us to explore CNN attention in SSL. We expect CNN encoders to maintain similar visual attention to transformers for recognition performance improvement with lower computational complexity and less memory consumption.
In this paper, we propose a CNN Attention REvitalization framework (CARE) to make CNN encoder attentive via transformer guidance. Fig. 1 (d) shows an intuitive illustration of CARE and compares it with other state-of-the-art SSL frameworks. There are two streams (i.e., C-stream and T-stream) in CARE where each stream contains two branches. C-stream is similar to existing SSL frameworks with two CNN encoders, two projectors, and one predictor. T-stream consists of two transformers, two projectors, and one predictor. T-stream takes CNN encoder features as input and improves feature attention via transformers. During the training process, we perform SSL in both streams simultaneously and use the T-stream output to supervise C-stream. The self-supervised learning in T-stream ensures attentive features produced by transformers are suitable for this SSL scenario. Meanwhile, we use the attention supervision on C-stream. This supervision enables both C-stream and T-stream to produce similar features. The feature representation of CNN encoders is improved by visual attention from transformers. As a result, the pre-trained CNN encoder produces attentive features, which benefits downstream recognition scenarios. Experiments on standard image classification, object detection, and semantic segmentation benchmarks show that the proposed CARE framework improves prevalent CNN encoder backbones to the state-of-the-art performance.

Related works
In the proposed CARE framework, we introduce transformers into self-supervised visual representation learning. In this section, we perform a literature survey on related works from the perspectives of visual representation learning as well as vision transformers.

Visual representation learning
There is an increasing need to learn good feature representations with unlabeled images. The general feature representation benefits downstream visual recognition scenarios. Existing visual representation learning methods can be mainly categorized as generative and discriminative methods. The generative methods typically use an auto-encoder for image reconstruction [63,49], or model data and representation in a joint embedding space [18,6]. The generative methods focus on image pixel-level details and are computationally intensive. Besides, further adaption is still required for downstream visual recognition scenarios.
The discriminative methods formulate visual representation learning as sample comparisons. Recently, contrastive learning is heavily investigated since its efficiency and superior performance. By creating different views from images, SSL obtains positive and negative sample pairs to constitute the learning process. Examples include memory bank [70], multi-view coding [58,61], predictive coding [30,60], pretext invariance [42], knowledge distillation [22,10] and information maximization [32]. While negative pairs are introduced in MoCo [29] and SimCLR [12], studies (e.g., BYOL [24] and SimSiam [13]) show that using only positive pairs are effective. Also, clustering methods [8,9] construct clusters for representation learning. The negative pairs are not introduced in these methods. Besides these discriminative methods focusing on image data, there are similar methods learning representations from either video data [66,26,33,67,45,64,65] or multi-modality data [1][2][3]. Different from these SSL methods, we use the transformer architectures to improve CNN encoders attention.

Vision transformers
Transformer is proposed in [62] where self-attention is shown effective for natural language processing. BERT [17] further boosts its performance via self-supervised training. The sequential modeling of transformer has activated a wide range of researches in natural language processing [57], speech processing [56], and computer vision [25]. In this section, we only survey transformer-related works from the computer vision perspective.
There are heavy researches on transformers in both visual recognition and generation. ViT [19] has shown that CNN is not a must in image classification. DETR [7], Deformable DETR [78] and RelationNet++ [16] indicate that transformers are able to detect objects with high precisions. SETR [76] brings transformers into semantic segmentation while VisTR [69] has shown transformers are able to perform video object segmentation. TrackFormer [40] introduces transformers into multiple object tracking. A general form of transformer is formulated in NLM [68] for video classification. Furthermore, transformers have been show effective in image generation [46] and image processing [11] scenarios. Examples include image super-resolution [72], video inpainting [74], and video captioning [77]. There are several emerging studies [15,10,4,71] on how to use selfsupervised learning to improve a transformer backbone. The learning paradigm for CNN encoders is adapted to the transformer without significant alteration. Different from existing methods that focus on learning a transformer encoder backbone with supervised or self-supervised learning. We explore how to use the transformers as guidance to enhance CNN visual attention. The pretrained CNN encoder benefits downstream recognition scenarios.

Proposed method
Our CARE framework consists of C-stream and T-stream. Fig. 2 shows an overview of the pipeline. We first illustrate the detailed structure of these streams. Then, we illustrate the network training process. The CNN encoder features are visualized as well for attention display.

CNN-stream (C-stream)
Our C-stream is similar to the existing SSL framework [24] where there are two CNN encoders, two projectors, and one predictor. The structures of the two encoders are the same, and the structures of the two projectors are the same. Given a training image x, we process it with a set of random augmentations to create two different augmented views. We feed these two views to C-stream and obtain corresponding outputs f 1 (x) and f 2 (x), respectively. Then, we compute a loss L c to penalize where · 2 is the 2 normalization, and the , is the dot product operation. As the inputs of C-stream are from one image, the outputs of C-stream are supposed to become similar during the training process.

Transformer-stream (T-stream)
The T-stream takes the output feature maps of the CNN encoders as its inputs, which are set in parallel to the C-stream. It consists of two transformers, two projectors, and one predictor. The structures of the projectors and the predictor are the same as those in the C-stream. The structures of two transformers share the same architecture, which consists of n consecutive attention blocks where each block contains two Multilayer Perception (MLP) layers with one multi-head self-attention (MHSA) layer in between. We mainly follow the design of [55] to construct the attention block in transformer as shown in Fig. 3. The input feature map (denoted as s) of an attention block is first processed by the a MLP layer for dimension mapping, and then passes the self-attention layer and finally the another MLP layer. MHSA consists of multiple attention heads that process the input features in parallel. In one attention head, as illustrated on the right of Fig. 3, the input feature map is mapped to the query feature (q), the key feature (k), and the value feature (k) via 3 different MLP layers w q , w k , and w v , respectively. As detailed in Eq. (2), the query q and key k are multiplied to form the content-based attention, and q and the position encoding p are multiplied to form the position-based attention.
where d k is the dimension of the query and the key. There are learnable parameters in the positional encoding module [46] to output p that follows the shape of s. In Eq.
(2), we perform matrix multiplication between q and p T , q and k T , and the softmax output and v by treating each pixel as a token [62] (i.e., for a feature map with c channels and spatial dimension of h × w, it forms h · w c-dimensional tokens and thus obtains a matrix of size h · w × c). Besides, we perform matrix addition between qp T and qk T . The output of the second MLP layer is added to the original input feature map s via a residual connection [27], and finally passes a ReLU activation layer.
In T-stream, the outputs of the two transformers are feature maps with 2D dimensions, which are then average-pooled and sent to the projectors and the predictor. We denote the outputs of T-stream as f 3 (x) and f 4 (x). Following the dissimilarity penalty in Sec. 3.1, we compute L t as follows: Besides introducing SSL loss terms in both streams, we use the T-stream output to supervise the C-stream. This attention supervision loss term can be written as: where the C-stream outputs are required to resemble the T-stream outputs during the training process. Note that the network supervision in Eq. 4 can not be simply considered as the knowledge distillation (KD) process. There are several differences from 3 perspectives: (1) The architecture design between ours and KD is different. In KD [31], a large teacher network is trained to supervise a small student network. In contrast, the CNN backbones are shared by two similar networks in our method. The different modules are only transformers, lightweight projectors, and predictor heads. (2) The training paradigm is different. In KD, the teacher network is typically trained in advance before supervising the student network. In contrast, two branches of our method are trained together from scratch for mutual learning. (3) The loss function in KD is normally the cross-entropy loss while we adopt mean squared error. During KD, supervision losses are also computed between feature map levels. While our method only computes losses based on the network outputs.

Network training
The proposed CARE consists of two streams and the loss terms have been illustrated above. The final objective function for network training can be written as: where λ is a constant value controlling the influence of the attention supervision loss. After computing L total , we perform back-propagation only on the upper branches of the C-stream and T-stream. Specifically in Fig. 2, the CNN encoder 1 , the projector 1 , and the predictor 1 are updated via the computed gradients in C-stream. Meanwhile, the Transformer 1 , the projector 2 , and the predictor 2 are updated via computed gradients in T-stream. Afterwards, we perform a moving average update [35,24,29] on the momentum CNN encoder 2 based on the CNN encoder 1 , on the momentum projector 1 based on the projector 1 , on the momentum transformer 1 based on the transformer 1 , and on the momentum projector 2 based on the projector 2 . We only use positive samples when training the network. As analyzed in [24], using momentum projectors and a predictor is shown important for self-supervised learning. These modules prevent CNN features from losing generalization abilities during the pretext training. Besides, we experimentally found that the momentum update is effective in preventing trivial solutions. In our network, we adopt a similar design in both streams to facilitate network training and obverse that using only positive samples does not cause model collapse. After pretext training, we only keep the CNN encoder 1 in CARE. This encoder is then utilized for downstream recognition scenarios.

Visualizations
Our CARE framework improves CNN encoder attention via transformer guidance. We show how encoders attend to the input objects by visualizing their attention maps. The ResNet-50 encoder backbone is used for visualization. We train this encoder for 200 epochs using only C-stream and the whole CARE framework, respectively. For input images, we use [51] to visualize encoder responses. The visualization maps are scaled equally for comparison. Overall, the attention of the encoder trained with CARE is more intense than that with C-stream, which indicates that T-stream in CARE provides effective supervision for CNN encoders to learn to attend to object regions. The T-steam helps CNN encoders adaptively choose to focus on local regions or global regions. For example, when global information is needed for classification, the CNN encoder learned by CARE will pay more attention to the whole object, as in the last column in (c), rather than a limited region, as shown in (b). On the other hand, when local information is sufficient for classification, the CNN encoder learned via CARE will pay more intense attention to the specific regions (e.g., the animals' heads in (c) on the first and second columns). The attention maps shown in the visualization indicate that the CNN encoder becomes attentive on the object region via transformer guidance in our CARE framework.

Experiments
In this section, we perform experimental validation on our CARE method. First, we introduce implementation details. Then, we compare our CARE method to state-of-the-art SSL methods on standard benchmarks, including image classification, object detection, and semantic segmentation. Furthermore, we conduct ablation studies to analyze each component of the proposed CARE method.

Implementation details
Image pre-processing. The training images we use during pretext training are from the ImageNet-1k [50] dataset. We follow [24] to augment image data before sending them to the encoders. Specifically, we randomly crop patches from one image and resize them to a fixed resolution of 224 × 224. Then, we perform random horizontal flip and random color distortions on these patches. The Gaussian blur, the decolorization, and the solarization operations are also adopted to preprocess these patches.
Network architectures. We use ResNet encoder backbones [27] (i.e., ResNet-50, ResNet-101, and ResNet-152) in our experiments. The architectures of the projectors and the predictors are the same and follow [24]. Each projector and predictor consist of a fully-connected layer with a batch normalization and a ReLU [44] activation, followed by another fully-connected layer. The transformer in the T-stream contains n attention blocks as shown in Fig. 3.
Network training process. We use the SGD optimizer with a momemtum of 0.9 during pretext training. The base learning rate is set as 0.05 and scaled linearly with respect to the batch size [23] (i.e., lr base = 0.05 × BatchSize/256). We start the pretext training with a warm-up of 10 epochs where the learning rate rises linearly from 10 −6 to the base learning rate (lr base ). Then, we use a cosine decay schedule for the learning rate without restarting it [39,24] to train the network. The momentum update coefficient of network parameters (denoted as τ ) is increased from 0.99 to 1 via a cosine design (i.e., τ = 1 − (1 − τ base ) · (cos(πt/T ) + 1)/2, where t is the current training step and T is the total number of training steps). We train CARE using 8 Tesla V100 GPUs with a batch size of 1024. The automatic mixed precision training strategy [41] is adopted for training speedup.

Comparison to state-of-the-art approaches
We compare feature representations of CNN encoders learned by our method and state-of-the-art SSL methods. Comparisons are conducted on recognition scenarios, including image classification (selfsupervised and semi-supervised learning configurations), object detection, and semantic segmentation.
Self-supervised learning on image classifications. We follow [29] to use standard linear classification protocol where the parameters of the encoder backbone are fixed and an additional linear classifier is added to the backbone. We train this classifier using SGD for 80 epochs with a learning rate of 0.2, a momentum of 0.9, and a batch size of 256. The ImageNet training set is used for the training and the ImageNet validation set is used for evaluation.    Table 2, where our CARE method also consistently prevails.
Semi-supervised learning on image classifications. We evaluate our CARE method by using a semi-supervised training configuration on the ImageNet dataset. After pretext training, we finetune the encoder by using a small subset of ImageNet's training set. We follow the semi-supervised learning protocol [24,12] to use 1% and 10% training data (the same data splits as in [12]) during finetuning. Table 3 shows the top-1 and top-5 accuracy on the ImageNet validation set. The results indicate that our CARE method achieves higher classification accuracy than other SSL methods under different encoder backbones and training epochs.
Transfer learning to object detection and semantic segmentation. We evaluate CARE's representations on the downstream object detection and semantic segmentation scenarios. We use the standard VOC-07, VOC-12, and COCO datasets [20,36]. We follow the standard protocol [29] to integrate the pretext trained CNN encoder into Faster-RCNN [48] when evaluating object detection results on VOC-07 and VOC-12 datasets. On the other hand, we integrate this encoder into Mask-RCNN [28] when evaluating object detection and semantic segmentation in COCO dataset. The ResNet-50 encoder is used in all the methods. All detectors are finetuned for 24k iterations using VOC-07 and   Besides, we further evaluate CARE's representation on the COCO dataset via a more powerful detector, the feature pyramid network (FPN) [37], and report the results in Table 5. We follow the same evaluation protocol introduced above. The detectors are trained with 1× schedule (90k iterations) for fair comparisons. Again, CARE trained for 200/400 epochs outperforms other state-ofthe-art SSL methods trained for 800/1000 epochs on object detection and semantic segmentation on the COCO dataset, which suggests that the CNN encoder in CARE is empowered by the attention mechanism of the transformer which supervises the CNN encoder in the pretraining.    [62] 66.68 learnable absolute [54] 72.01 learnable relative [52] 72.06

Ablation studies
In our CARE method, visual attention is explored via transformers to supervise C-stream. We analyze the influence of attention supervision by using different values of λ in Eq. (5). Also, we analyze how the number of attention blocks and the positional encoding affect feature representations. Besides, we further take an investigation of the sequential and parallel design of T-stream. We use the ResNet-50 as encoder backbone and the number of training epochs is set to 100. The top-1 image classification accuracy on ImageNet via SSL is reported to indicate feature representation effectiveness.
Supervision influence λ. We study how the attention supervision term L att in Eq. (5) affects feature representation by using different values of λ. Table 6 shows the evaluation results. When we set λ as 0, the CNN encoder is learned without attention supervision. In this case, the performance decreases significantly. When we increase the value of λ, the attention supervision increases as well. We observe that λ = 100 achieves the best performance and adopt this setting in the overall experiments.
Number of attention blocks. We analyze how the capacity of transformer in T-stream affects the recognition performance. We set n = [2, ..., 6] to constitute five transformers in T-stream with increasing capacities. Then, we report the recognition performance of the corresponding CNN encoders in Table 7. When n is larger, the transformer capacity increases and stronger visual attention are explored. This attention supervises C-stream to improve the encoder. However, the recognition performance drop when n = 6. This may be due to the broken balance between the attention loss L att and the original SSL loss L c . In our experiment, we set n = 5 in our CARE method.
Positional encoding. We analyze how positional encoding affects the final performance. Numerous positional encoding settings [62,54,52] are adopted for analysis in Table 8. Without positional encoding, the performance decreases significantly. Meanwhile, the conventional fixed sine-cosine encoding setting is not suitable for CARE. We experimentally find that using learnable parameter configuration in positional encoding improves recognition performance and adopt [52] in CARE.
Sequential design v.s. parallel design. Experimental results on training ResNet-50 with 100 epochs indicate that parallel design is effective than sequential design (i.e., 72.02% v.s. 69.32%). This is because during the sequential training process, both the CNN encoders and the transformers are optimized together rather than the CNN encoders themselves. This prevents attention supervision from training the CNN encoders thoroughly.

Concluding remarks
Transformers have advanced visual recognition via attention exploration architectures. In selfsupervised visual representation learning, studies emerge to utilize transformer backbones for recognition performance improvement. This indicates that visual attention explored by the transformers benefit SSL methods. Motivated by this success, we investigate how to explore visual attention effectively to benefit CNN encoders in SSL. We propose CARE to develop a parallel transformer stream together with the CNN stream. The visual attention is thus explored via transformers to supervise the CNN stream during SSL. Although the limitation occurs that more computational cost is spent on the SSL and attention supervision loss term, the learned CNN encoder becomes attentive with transformer guidance and does not consume more costs in downstream tasks. Experiments on the standard visual recognition benchmarks, including image classification, object detection, and semantic segmentation, indicate that CARE improves CNN encoder backbones to a new state-of-the-art performance.