{'fraction_non_alphanumeric': 0.028796259337400924, 'fraction_numerical': 0.00672863089467982, 'mean_word_length': 4.939045038943448, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Feature Extraction with Apparent to Semantic Channels for Object Detection

This paper focuses on using traditional image processing algorithms with some apparent-tosemantic features to improve the detection accuracy. Based on the optimization of Faster R-CNN algorithm, a mainstream framework in current object detection scenario, the multi-channel features are achieved by combining traditional image semantic feature algorithms (like Integral Channel Feature (ICF), Histograms of Gradient (HOG), Local Binary Pattern (LBF), etc.) and advanced semantic feature algorithms (like segmentation, heatmap, etc.). In order to realize the joint training of the original image and the above feature extraction algorithms, a unique network for increasing the accuracy of object detection and minimizing system weight called Multi-Channel Feature Network (MCFN) is proposed. The function of MCFN is to provide a multi-channel interface, which is not limited to the RGB component of a single picture, nor to the number of input channels. The experimental result shows the relationship between the number of additional channels, performance of model and accuracy. Compared with the basic Faster R-CNN structure, this result is based on the case of two additional channels. And the universal Mean Average Precision (mAP) can be improved by 2%-3%. When the number of extra channels is increased, the accuracy will not increase linearly. In fact, system performance starts to fluctuate in a range after the number of additional channels reaches six.


Introduction
In the field of target detection, Faster R-CNN [1] algorithm has excellent operation accuracy because of the low coupling degree of its internal modules, so it can optimize the structure of various application scenarios. This is also different from YOLO series [2] and SSD series [3], because the latter pay more attention to model integration and lightweight.
Since the convolution neural network (CNN [4], [5]) has been widely used in 2015, Faster R-CNN is combined with excellent convolutional networks such as residual network [6], and the introduction of Region Proposal Network (RPN) Local modules make the whole system be divided into three layers. The bottom layer is the feature extraction stage. The neural network algorithms used include ResNet 101, VGG-16 and so on. The second layer is the Region Proposal Network (RPN), This is also the most obvious difference between Fast R-CNN [7] and R-CNN [8]. The top layer roughly retains the framework of the previous generation, including the bounding box regression and the softmax layer. The Faster R-CNN algorithm still needs the optimization of object detection because the average accuracy of some well-known datasets (such as VOC 0712) are still less than 80% based on the analysis of experimental data. Therefore, this paper is dedicated to algorithm optimization from the change of the structure of the input end of the algorithm. This structural optimization is based on the input of different image channels, and different channels can provide different semantic information, so that the bounding box can be framed more accurately in the RPN structure. Image processing algorithms are gradually being replaced by neural networks in the field of object detection, but this does not mean that traditional algorithms have been abandoned, whether the traditional algorithm can be combined with neural network remains to be explored. Fig. 1 shows a brief process of Faster R-CNN with multi-channel. The multi-channels and side branch in this figure are the optimization goals of this article.

Details of Apparent-to-Semantic Features
The extra image channels in Fig. 1 all belong to apparent-to-semantic features. For example, semantic segmentation maps, heatmap, HOG, ICF, etc. Apparent semantic features mean defining object categories from different perspectives. For example, semantic segmentation considers which category belongs to in the pixel level, while textures and edges tend to identify the detailed shape of each object. But we need to solve one problem that which kind of feature is effective and how it actually works to improve the R-CNN-based detectors. To answer this question, this paper explains from the following three aspects: • Firstly, we integrate extra features as input channels into CNN-based detectors. To investigate apparent-to-semantic channels, extensive experiments are carried out on two datasets (PASCAL VOC 0712 [9] and COCO [10]), and inspired by the excellent work of J. Mao et al. [11], the feature extraction network is ResNet 101. Besides, the average recognition accuracy of benchmark is 77.61% [12] in this paper. • Then, we experimentally analyzed both advantages and disadvantages of different channels of feature.
Specifically, we quantify the improvement brought by different channel features and provide insight into the error sources. In Fig. 2, this figure shows the performance of additional channels in the KITTI dataset [13]. From the results, the accuracy of segmentation [8], [14], heat map [15], [16] and edge [17] is better than benchmark. Relatively speaking, the positive contribution rate of ICF [18], [19], HOG [20] and LBF [21] to the benchmark is less than 1%, and even the negative contribution rate appears. This is because of the low correlation, which cannot further improve the pixel level features.

Multi-channel Joint Learning
This paper optimizes based on Faster R-CNN and residual network proposed by Kaiming He, et al. Faster R-CNN is generally considered to have three layers of substructures.
In the substructure system of Faster R-CNN, what this paper considers is to design a multi-channel structure of the feature extraction stage. As shown in Fig. 3. This structure still retains the original picture input channel, and uses 1 × 1 and 3 × 3 convolution kernels for convolution (conv_1 to conv_4). The convolution layer designed for additional channels is a fully convolution network. Multi-channel images are pre-processed before the input, and images from multiple channels of the same image share the same information of location and category table.
Different image processing algorithm focuses on different point, such as texture and segmentation. Texture focuses more on the content of the category, and segmentation focuses on the interpretation of boundaries between different categories. This is also the problem that this paper needs to solve that through a large number of comparative experiments, which combination method is used to improve the accuracy of target detection, in other words, which kind of image processing methods can better be used with target detection.
Integrating channel features in the network can boost our detector working on images of both low resolution and high resolution. With these channel capabilities, we can close most of the gap between resolutions without introducing the need to enlarge the input image and drive the heavy computational costs of the latest technology. Compared with the basic Faster R-CNN, the computational cost of the brute force integration method is high. Therefore, we propose a multi-channel feature extraction algorithm. The framework is illustrated in Fig. 3. As shown, our system consists of four components: the body network of origin images, the multi-channel feature network (MCFN) of extra channels, the regional proposal network (RPN), and the fast R-CNN network of final detection tasks.
Body Network As shown in Figure 3, the body network on the left side means the original part that this paper referred from Faster R-CNN. The body network takes the raw image of shape 3 × H × W, as its input, and outputs several activation maps. In this paper, the body network is a ResNet 101 network initialized with the pre-trained weights. ResNet 101 contains five layers of convolution. The results obtained by the image through conv 1, conv 2_x, conv 3_x, and conv 4_x will become the input of RPN and ROI pooling, respectively, and the generated results will be passed to the next structure through conv 5_x and Average pooling.

Multi-channel feature network (MCFN)
The MCFN directly takes the aggregated activation maps to generate the predicted channel feature maps through fully convolution structure. The feature map obtained from the original image is summarized into MCFN after hierarchical up-sampling. MCFN is composed of multiple independent full convolution networks. The output after conv 4_x will be passed to the RPN and Fast R-CNN together with the output of CFN. The hybrid feature map acts as a significant role in the whole framework, which must be carefully trained. MCFN needs to be trained separately, considering model performance, single-threaded operations are required. Secondly, the body network (conv 1_x to conv 4_x) and MCFN are jointly trained to generate the hybrid feature map. Then, Fast R-CNN is trained separately with the same residual network. conv 5_x and the average pooling layer are added to perform bounding box regression. Implement of RPN and Fast R-CNN As shown in Fig. 3, below the body network and MCFN, the implement of RPN and Fast R-CNN also refers the origin part from Faster R-CNN. We use the same structure for RPN and Fast R-CNN as proposed in [1]. RPN and Fast R-CNN now take both convolution activation map by MCFN in the ResNet network and the feature map from the body network as the inputs. The proposals generated by RPN are then fed into Fast R-CNN to perform final detection. In this process, faster r-cnn does not have the process of back propagation, and the process of updating the weight is estimated according to the relative position of the bounding boxes and the optimization of the loss function.
In summary, our changes to MCFN are based on the structural optimization of the algorithm of Faster R-CNN, which is also the main idea that proposed in this article. And this article will conduct a comparative analysis through experimental data, show the results after adding different additional channels, and give an analysis of its reasons.

Single Extra Channel of Feature Extraction
The work of J. Mao et al. [11] is targeted at the application scenario of the pedestrian dataset KITTI, and an additional single-channel optimization experiment is designed on this basis. In this experiment, an additional single-channel experimental design is first performed on a dataset in a more general scenario. The purpose of this is to set a reference value and to obtain a preliminary expected range for different algorithms through experimental results.
As show in Table 1 and Table 2. This idea is confirmed in the previous pedestrian detection, so this need to be carried out in a general target detection scenario. Synchronous experiments are a prerequisite for subsequent experiments. The reason is that the structure of the algorithm for pedestrians has not changed structurally after the abstraction of specific pedestrian detection scenes to general target detection. The recognition categories are more abundant, and the recognition image data set has also changed. The two tables show together that it is effective and feasible to add an additional channel without changing the overall experimental structure. By using two different datasets, the obvious gained results could be detected easily.
The reason for their effectiveness is that the additional channels can provide enough boundary information or semantic information through layer-by-layer upsampling, which can more accurately frame the starting point during RPN and frame regression and the regression function will also improve the results. In the two selected datasets, the experimental results of adding segmentation and heatmap channels outperform than other algorithms. It is also worth noting that not all channels will necessarily improve, such as adding LBP and ICF channels. This is because the loss function is related to the frame regression of the obtained feature map. For example, LBP pays more attention to the texture results of the image, but the design of the loss function pays more attention to the boundary, which results in no actual improvement in the results.

Multi-extra-Channel of Feature Extraction
Designing a dual-channel algorithm on the basis of an additional single-channel experiment is the most important step. Here it involves the choice of the algorithm combination and the priority. As show in Table 3. This table explains the feasibility of the network structure optimization proposed in section of proposal. After comparing the experimental results of several sets of additional dual channels, it can be concluded that under the same experimental environment, the results can be improved by increasing the number of channels and selecting a better performing algorithm. From the comparison of (0, 1, 2, 3), we can see the different results of different combinations of segmentation and other algorithms. The comparison of (3, 4) is for experiments on feature selection algorithms. (1,5) and (5,6) also compare the experimental results of adding channels on the basis of ICF and Edge. Different algorithm combinations are not randomly selected. What needs to be considered is whether algorithm A and algorithm B can provide image semantic information in the target detection scenario. If it is just an RGB-level matrix, then it will only increase the depth of the convolution rather than provide accurate position information. In addition, can the two algorithms operate in parallel? Because the image information provided by the two algorithms is aggregated after upsampling by the network to form more layers of input features. Such feature maps are layer-by-layer and the matrix operations may focus on the same area at the same time, but get different classification results. This also leads to errors.
In addition, when the number of channels increases, the relationship between recognition accuracy and algorithm performance is also worth considering. The curve in Fig. 4. (a) represents the relationship between second per figure (spf), hours per epoch (h/epoch), and the number of channels after the number of channels has been increased to 10. Fig. 4. (b) shows the relationship between the number of channels and accuracy. From this experimental result, it can be seen that when the number of channels is continuously increased, the burden of model training will be increased, and the recognition rate will be reduced. The recognition accuracy obtained in this way is not worth doing. The curves from Fig. 4. (c) to Fig. 4. (f) show the change of the mIoU value after adding different additional channel numbers. The value of mIoU generally reflects the gap of semantic information such as segmentation and heat map, and can intuitively reflect the contribution to the accuracy of object recognition in different situations. In (c), a single segmentation channel is added to the experiment, in (d), a single channel of heat map is added to the experiment, while in (e), two channels are added, segmentation and heat map. In (f), two channels are also added, both of which are segmentation graphs. The following is an overall evaluation of the mIoU experiment results. From the comparison between (c) and (d), The influence gap between single channels on mIoU is more obvious, and the segmentation map will have obvious advantages, because the segmentation map will generate stronger semantic information, and the boundary information included in the regression operation will be more. It can be seen from the two groups, {c, e} and {d, e}, on the basis of single channel, it is worth trying to combine two single channels for multi-channel calculation. This includes the fusion calculation of multiple information and the redundancy, complementarity and rejection of calculation parameters. As can be seen from (e) and (f), the comparison experiment between different combinations of multiple channels is also an important work, which shows that it is necessary to choose some of these combinations that have more advantages and can explain the direction of the reason.

Conclusion
This paper combines traditional apparent algorithms and advanced semantic algorithms to implement the framework optimization of target detection algorithms based on Faster R-CNN.
The experimental results are discussed in two aspects. The first aspect proves that without changing the internal structure of the network, only changing the mode from the input can improve the detection accuracy. There will be a difference in improvement when using different image processing algorithms. For example, segmentation and heatmap are 3% better than ICF and LBP. This also reflects the advantages of semantic information in feature extraction.
The second aspect shows that by implementing the form of multi-channel additional side branches, the images obtained by these algorithms and the original image are taken as inputs from different dimensions, and the final hybrid feature map obtained by parameter screening has certain optimization effects. For example, the channels combining segmentation maps and heat maps have an average recognition accuracy improvement of 3.74% compared to traditional Faster R-CNN. Meanwhile, when the number of experimental channels increases, it does not necessarily result in a significant increase of accuracy. In contrast, the performance of the model will increase dramatically. This also reminds us that the algorithm performance is also a point to consider optimization.