{'fraction_non_alphanumeric': 0.030024635598439747, 'fraction_numerical': 0.010367481010059536, 'mean_word_length': 4.790490341753343, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ReplaceBlock: An improved regularization method based on background information

Attention mechanism, being frequently used to train networks for better feature representations, can effectively disentangle the target object from irrelevant objects in the background. Given an arbitrary image, we find that the background's irrelevant objects are most likely to occlude/block the target object. We propose, based on this finding, a ReplaceBlock to simulate the situations when the target object is partially occluded by the objects that are deemed as background. Specifically, ReplaceBlock erases the target object in the image, and then generates a feature map with only irrelevant objects and background by the model. Finally, some regions in the background feature map are used to replace some regions of the target object in the original image feature map. In this way, ReplaceBlock can effectively simulate the feature map of the occluded image. The experimental results show that ReplaceBlock works better than DropBlock in regularizing convolutional networks.


Introduction
Convolutional neural networks (CNNs) have been widely and successfully used in various computer vision tasks [Krizhevsky et al., 2012;He et al., 2017]. In recent years, various deep CNN architectures have significantly improved the accuracy of image classification, such as ResNet [He et al., 2016] and InceptionNet [Szegedy et al., 2016]. Deep CNNs usually have a large number of parameters, which can very easily lead to over-fitting of the network. Therefore, it is worthy of developing regularization methods to alleviate the problem of over-fitting for CNNs. However, a recent study shows that traditional Dropout is less effective for CNNs because of the under-drop problem [Ghiasi et al., 2018;Zeng et al., 2020]. To address this issue, DropBlock [Ghiasi et al., 2018] developed a structured form of Dropout for CNN models, where contiguous regions of feature maps are dropped instead of individual elements in the feature maps.
Apart from the regularization approaches, we investigate the effect of attention in CNNs. The significance of attention has been studied extensively in the previous literature [Jaderberg et al., 2015;Woo et al., 2018;Yang et al., 2021]. The at- tention mechanism can distinguish between the target object and background of an image, which is widely used in weakly supervised object localization (WSOL) [Zhou et al., 2016;Choe and Shim, 2019;Yin et al., 2021]. WSOL aims to identify the object's location in a scene only using image-level labels, not location annotations. This is consistent with our goal of using attention, only using image-level labels, to obtain an image with only the background (hereinafter called background-only image) through the attention mechanism. To date, the general practice of regularization methods related to Dropout [Srivastava et al., 2014] has been to drop some of the features on the feature map directly, i.e., to erase the contiguous regions or individual pixels on the feature maps by setting them zero during the training phase, to encourage the models look elsewhere for evidence to fit the data. In WSOL, Dropout is usually combined with attention mechanism [Choe and Shim, 2019;Yin et al., 2021], erasing the most discriminative region on the feature map by zeroing that region during the training phase. This prevents the model from relying solely on the most discriminative part for classification and encourages it to learn the less discriminative part as well. It was demonstrated in CamDrop [Wang et al., 2019] that erasing the most discriminative part is effective to capture the full extent of object and obtain a more accurate object localization. However, none of these methods can simulate the occluded image in a natural way. Because in the real world, it is usually a specific object that occludes the target object, not a 0-1 mask. Although the attention mechanism can distinguish the target object and background of an image, unfortunately, the current attention technique rarely utilizes the information of the background region of an image.
From previous methods, we conclude that erasing the contiguous regions of the feature map can effectively alleviate the over-fitting of the model. But unlike the previous approach of direct dropping, in this paper, we propose a novel and efficient regularization method, ReplaceBlock, as shown in Figure 1. Utilized the ability of the attention mechanism to distinguish the target object and background of input images, we can obtain feature maps with only background information. ReplaceBlock replaces a small region of the target object in the original image feature map with background-only feature map blocks to simulate the feature map where the target object in the image is partially occluded. Specifically, a spatial self-attention map is obtained from the final feature map of the network. Based on the spatial self-attention map, we produce the first component of the ReplaceBlock, i.e., the target class object drop mask (TC-DM). We obtain this TC-DM by thresholding the spatial self-attention map. During the training, we first use the TC-DM to hide the objects to be recognized in the original image, and then use the model to obtain a deep feature map with only irrelevant background information. Also based on the spatial self-attention map, we produce the second component of the ReplaceBlock, i.e., the replacement region sampling map (RR-SM). The RR-SM is used to obtain the contiguous regions in the original image feature map that need to be replaced (The discriminative regions in the deep feature map of the original image have a higher probability of being replaced).
We summarize our contribution as follows: • We propose a novel regularization method, which can use the background information to effectively simulate the deep feature map of the occluded image, improving the generalization ability of the model.
• Our method replaces parts of the most discriminative region, which can help the model capture the full extent of the object, resulting in cleaner background-only images.
• Compared with DropBlock, ReplaceBlock significantly improves the performance in image classification and fine-grained image recognition tasks.

Related Work
This section summarizes two deep learning topics relevant to this work: (1) dropout-based regularization method and (2) attention mechanisms.

Dropout-based regularization method
In an attempt to apply a structured form of Dropout to the convolutional layer, Tompson and Jonathan [Tompson et al., 2015] proposed SpatialDropout that randomly drops partial channels of a feature map, rather than dropping each pixel. Cutout [DeVries and Taylor, 2017]  proposes an augmentation strategy: patches are cut and pasted among training images where the ground truth labels are also mixed proportionally to the area of the patches. Different from CutMix, which mixes two images and their labels to augment the input data, ReplaceBlock uses background information to simulate feature maps of occluded images.

Attention mechanism
Attention mechanism [Tsotsos, 1990] have been proven helpful in a variety of computer vision tasks, such as image classification [Hu et al., 2018a] and face recognition [Hu et al., 2018b]. The squeeze-and-excitation module [Hu et al., 2018b] [Choe and Shim, 2019] propose an Attention-based Dropout Layer, which utilizes the self-attention mechanism to hide the most discriminative part from the model for capturing the integral extent of object. However, the key difference from them is that we locate the target object with the aim of obtaining an image with only background information to establish the basis for the replacement operation.

ReplaceBlock
Here, we take replacing the input feature map of the final convolutional layer as an example to introduce ReplaceBlock, as shown in Figure 2. Let G bb ( ) be all the convolutional layers of the model excepting the final conv layer, and G f inal ( ) be the final conv layer of the model. Given the input image x, we have its feature maps: and where F ∈ R H×W ×C be a feature map. Note that C denotes the channel number, H and W are the height and the width of the feature map, respectively. For simplicity, the mini-batch dimension is omitted in this notation.
To obtain the TC-DM, we perform CAM [Zhou et al., 2016] on the F pf to derive the 2D spatial self-attention map M mask ∈ R H×W . We observed that the M mask usually highlights the most discriminative regions of the target object and suppresses less discriminative ones. We use a simple

Replace
Replaced deep feature map Figure 2: ReplaceBlock diagram. The blue arrows indicate the method of generating the background-only image and the background information feature map. The "Attention map" is generated from the deep feature map by CAM. Note that "Shallow layer" and "Deep layer" in the figure share parameters in both calculations. thresholding technique to segment the self-attention map to obtain the TC-DM: B img . We first set a drop threshold value that is 20% of the max value of the M mask . Then, we generate the TC-DM: B img ∈ {0, 1} H×W by setting each pixel to 0 if it is larger than the drop threshold, and 1 if it is smaller. That is, the drop mask has 0 for the region to be recognized and 1 for the background region. The drop mask is applied to the input image by element-wise multiplication. In this way, we can hide the region of the target object in the input image. Calculation of the feature map for the irrelevant objects and background: where is the general element-wise multiplication, F BG pf denotes the feature map of the image with only irrelevant objects and background. To use the background information more effectively, we randomly shuffle F BG pf in the spatial dimension, as shown in Figure 3. Using the random shuffle strategy allows the ReplaceBlock to fetch the background at any position. Note that F BG pf and F pf are calculated from the same G bb ( ) in each forward propagation. Then, we use the M mask sampling to get M sample : M sample ∼ B(M mask ). Using M mask makes it more likely to drop regions with stronger discriminability. Because there is a strong correlation between the neighboring pixels on the convolutional layers, and these adjacent pixels have similar information, we use a similar approach to DropBlock to obtain a structured form of drop mask B f eature from M sample . B f eature drops the contiguous regions of feature maps rather than the individual pixels, as illustrated in . background feature map F BG pf , formally as : where 1 is a binary mask filled with ones. Finally, the output feature of the final conv layer is as follows: In this way, we replace a small portion of the most discriminative regions with feature maps of irrelevant objects and background, encouraging the model to adapt to the situation where the most discriminative regions of the image are partially occluded.

Background-only image after being masked
ReplaceBlock inputs the masked image (B img x) directly into the model to extract features. To make more effective use of background information, we have tried to use image inpainting technology to repair B img x and obtain a . background-only image without black regions to get the complete background information. The experiment found that the training time cost is too large to obtain a credible repair result, and it is not feasible. How to better process B img x is the current limitation of ReplaceBlock, and it is also one of the concerns of our next step research.

Experiments
In this section, we first describe our experiment dataset and experiment settings, then conduct an ablation experiment to demonstrate the relation of the background information to the proposed ReplaceBlock to the performance. Next, we compare our approach with some of the most popular current regularization methods. Finally, we visualize the selfattention map obtained by our proposed method and the vanilla method.

Experiment Settings
Dataset. We evaluate the performance of our method using to implement all our experiments. During training, we used the standard SGD optimizer with a momentum of 0.9 to train all the models. The weight decay is set to 4E-5 always. The cosine learning schedule [He et al., 2019] with an initial learning rate of 0.01 is adopted. All experiments are conducted on an NVIDIA GeForce GTX 1080Ti (11 GB) GPU. We use ResNet-50 as the backbone. The images are cropped to 224 × 224 as the input. Random horizontal flipping and random cropping are used for data augmentation. Where to apply ReplaceBlock. For a fair comparison with DropBlock, we follow the setting of DropBlock [Ghiasi et al., 2018], applying ReplaceBlock in the last two Groups of ResNet-50. In a Group, apply ReplaceBlock after both convolution layers and skip connections.   Generating B f eature requires two hyperparameters, keepprob and block-size. keep-prob controls the drop rate, and block-size controls the size of the dropped region. In this paper, we refer to the setting of DropBlock with keep-prob=0.9 and block-size=3.

Ablation Studies
To demonstrate the effect of different background information on the performance of the proposed ReplaceBlock, we set different thresholds to the CAM to obtain TC-DM, resulting in different background-only images. First, we visualize the image masks obtained by different thresholds in Figure 5. The region dropped by CAM(5%) includes not only the object to be recognized, but also other objects in the image, as shown in the second column of Figure 5; although the object to be recognized is the dog, the region of the boat in the image is also dropped. CAM(60%) only drops the most discriminative regions in image, which has the problem of under-dropping. From Figure 5, it can be seen that the best image mask is obtained by CAM(20%).
Then, we use ResNet-50 as a baseline to test the performance variation of the proposed ReplaceBlock method when different image masks are applied. In this experiment, we adopt three image mask obtaining approaches described above. As shown in Table 1, the model using CAM(20%) has the best performance, which corresponds to the masked image shown in Figure 5. This indicates that the performance of ReplaceBlock was related to the background information used as a replacement and that a cleaner background image lead to a better performance improvement. The ReplaceBlock is the same as the DropBlock when using a background image with all pixel values of 0.
To demonstrate the impact of RR-SM on the performance of ReplaceBlock, we obtain drop masks by random sam-pling and RR-SM sampling, respectively. As shown in Table  2, using RR-SM gets better results, but the improvement is smaller. This may be because when there are enough training epochs, the randomly generated drop mask also contains a large number of samples that drop the more discriminative regions. To explore whether ReplaceBlock leads to 'background bias' (the model does not see the target object, but use backgrounds in classification), we set up an experiment to train the model in two ways: (1) applying the ReplaceBlock at each training step, and (2) applying the ReplaceBlock alternately (the current training step applies the ReplaceBlock, and the next training step does not apply the ReplaceBlock). As shown in Table 3, the performance of applying ReplaceBlock all the time is much better than that of applying ReplaceBlock alternately. We think there are two possible reasons: (1) the drop rate set by ReplaceBlock is small (10%), the discriminative region of the feature map is still dominant, (2) Different images and different positions of the same image both contain different backgrounds, making it difficult to overfit the model to a particular background, so ReplaceBlock can simulate the feature map of the occluded image well without causing 'background bias'.  Ghiasi et al., 2018]. As can be seen from Table 4 and Table 5, adding the DropBlock has already improved the classification accuracy by more than 2.3%. The Cutout has a lower performance than DropBlock, which indicates that dropping at the feature map is better than dropping directly on the input image. However, when the proposed Re-placeBlock is considered, we achieve the best results. We argue that the advantages of the proposed Replace-Block over DropBlock are two-fold. First, the DropBlock Table 4: Top-1 and Top-5 accuracies (%) for ResNet-50 model with different regularization methods, on Mini-ImageNet datasets. The ReplaceBlock module follows the best settings in the ablation studies, the other regularization methods follow the settings in their papers.

Method
Top-1 acc.   Table 6, we can draw a similar conclusion as Table 4 and Table 5. ReplaceBlock achieves the best results on these two FGVC datasets. Compared with DropBlock, our method still gains 0.84% and 0.49%, respectively.

Visualize the heatmap
To validate whether ReplaceBlock can help the model capture a complete target object, we show the heatmaps obtained by different methods. Figure 6 visualizes the heatmaps of our method and the vanilla model on the Mini-ImageNet datasets.
From the results, we consistently observe that model with ReplaceBlock captures more complete target object than the vanilla model. The vanilla model usually captures only the most discriminative parts. For example, as seen from the leftmost sample in the Figure 6, the heatmap extracted from the ReplaceBlock vanilla ReplaceBlock vanilla Figure 6: Qualitative evaluation results of ReplaceBlock on Mini-ImageNet. We compared our method and the vanilla model. The left image in each figure is input image. The middle image is heatmap and the right image shows the overlap between the input image and the heatmap. vanilla model only highlight the neck of birds. Contrarily, the model with ReplaceBlock covers not only the neck, but also the entire part of the bird, from neck to tail. In addition, from the third sample in Figure 6, the vanilla model focuses only on the eyes of dogs, whereas the model with ReplaceBlock localizes the entire region of the dog. Based on the above analysis, ReplaceBlock can help the model capture the full extent of target object and obtain a cleaner background-only image to improve the model's efficiency in utilizing background information.

Conclusions
In this paper, we present a novel regularization method for CNNs, named as ReplaceBlock. ReplaceBlock is a form of structured replacement that utilizes irrelevant background information to replace spatially correlated region. Replace-Block is an auxiliary module which is applied only during training stage. And in the testing stage, ReplaceBlock is deactivated. Our ReplaceBlock inherits the advantages of Drop-Block in structuring the erasure information, but Replace-Block uses a replacement operation instead of direct dropout to better simulate the case of the occluded image, resulting in a more diverse feature map. Meanwhile, ReplaceBlock can induce the CNN classifier to learn the entire extent of the object and obtain cleaner background-only images, which provides a guarantee for using background information. Experiments in Mini-ImageNet classification and fine-grained image classification demonstrate the effectiveness of our Re-placeBlock. We hope that our work will promote the study of regularization methods for the application of irrelevant background information.