{'fraction_non_alphanumeric': 0.0369404451430817, 'fraction_numerical': 0.016148047586724303, 'mean_word_length': 4.690587517788169, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Towards Versatile Pedestrian Detector with Multisensory-Matching and Multispectral Recalling Memory

Recently, automated surveillance cameras can change a vis- ible sensor and a thermal sensor for all-day operation. How-ever, existing single-modal pedestrian detectors mainly focus on detecting pedestrians in only one specific modality ( i.e., visible or thermal), so they cannot cope with other modal inputs. In addition, recent multispectral pedestrian detectors have shown remarkable performance by adopting multispec- tral modalities, but they also have limitations in practical applications ( e.g., different Field-of-View (FoV) and frame rate). In this paper, we introduce a versatile pedestrian detector that shows robust detection performance in any single modality. We propose a multisensory-matching contrastive loss to reduce the difference between the visual representation of pedestrians in the visible and thermal modalities. More-over, for the robust detection on a single modality, we design a Multispectral Recalling (MSR) Memory. The MSR Memory enhances the visual representation of the single modal features by recalling that of the multispectral modalities. To guide the MSR Memory to store the multispectral modal con- texts, we introduce a multispectral recalling loss. It enables the pedestrian detector to encode more discriminative fea- tures with a single input modality. We believe our method is a step forward detector that can be applied to a variety of real- world applications. The comprehensive experimental results verify the effectiveness of the proposed method.


Introduction
Pedestrian detection is a widely studied task in computer vision, and it has been applied to massive real-world applications (Dalal and Triggs 2005;Benenson et al. 2014;Chi et al. 2020). When detecting pedestrians, many methods have utilized a single camera sensor. Traditional methods commonly adopt a visible modality (e.g., RGB), because this modality contains various color information about pedestrians (Zhang et al. 2021). In addition, recent works have adopted a thermal modality, because this modality is robust to various challenging environments, such as low illumination (Li et al. 2019b) and background clutter (Song et al. 2019).
Since the two modalities have different properties, recent surveillance cameras automatically switch the input modality between the visible modality and the thermal modality depending on the situation for around-the-clock operation (Wu et al. 2017). In general, visible modality is adopted in day time where color information is well observed and thermal modality is used in cloudy weather or dark nights where pedestrians cannot be seen but heat information is detected. Thus, it is necessary to build a network that is robust to changes in the input modality to automatically detect pedestrians. However, previous single-modal methods have a limitation in that they mainly focused on detecting pedestrians in only one specific modality (visible or thermal), so that they cannot effectively cope with other modal inputs (Wu et al. 2020b;Kieu et al. 2020). For instance, thermalbased detectors cannot handle visible modality due to the significant differences between the two modalities (Li et al. 2019a). Some efforts of person re-identification task have been made to cope with changes in input modality (Wu et al. 2017(Wu et al. , 2020a. However, they need additional modal-specific networks, and the users should know the input modality type in advance to switch modal-specific networks. Moreover, some recent works have studied on multispectral pedestrian detection which utilizes both visible modality and thermal modality simultaneously (Hwang et al. 2015;Zhang et al. 2019;. Since this task can utilize complementary visual information of the two modalities, multispectral pedestrian detectors can encode more discriminative features (Hwang et al. 2015), showing remarkable performance. However, multispectral pedestrian detection is somewhat difficult to be applied to practical applications for several reasons. For example, the multispectral pedestrian detectors need an image pair of two modalities that have the same Field-of-View (FoV) and same frame rate (Zhang et al. 2019) in the inference phase. Also, the computational cost is much higher than the single modal detectors.
Based on the above discussions of the single modality and multispectral modalities, it is essential to build a versatile pedestrian detector that overcomes the above limitations. We aim to build a pedestrian detector that shows robust performance while freely receiving the visible modality and thermal modality. To this end, we consider following two issues: (1) how to cope with a single input regardless of the input modality type and (2) how to enhance the visual representation of a single modality for the robust pedestrian detection.
In this paper, we propose a novel pedestrian detector by addressing the raised two issues. First, in order to make the modality-agnostic detector, we propose a multisensorymatching contrastive loss. The multisensory-matching contrastive loss guides the two modal pedestrian features to be similar by referring to the counterpart modal features. It makes the pedestrian features close to each other and separates pedestrian features and background features. Through the multisensory-matching contrastive loss, the proposed detector can encode modality-agnostic pedestrian features. Second, to improve the visual representation of a single modality, we design a Multispectral Recalling (MSR) Memory. The role of the MSR Memory is to enhance the single modal pedestrian features by addressing relevant information to recall the visual appearance of multispectral modalities that contain richer visual information. In order for the MSR Memory to memorize the multispectral contexts, we introduce a multispectral recalling loss. As a result, more discriminative pedestrian features can be encoded from any single modal input through the MSR Memory.
Our work is to design a versatile pedestrian detector that shows robustness for any single modality by recalling the multispectral information. Therefore, we would like to claim that the proposed method is a step forward method that can be applied to various real-world applications, such as recent surveillance cameras that automatically switch between the two modalities. Through the comprehensive experimental results and visualization results, we demonstrate the effectiveness of the proposed pedestrian detector with the multisensory-matching and the MSR Memory.
The major contributions of this work can be summarized as follows: • We introduce a multisensory-matching contrastive loss in order to guide the network to have similar visual representations about pedestrian regardless of the modality. • We devise MSR Memory that can recall the visual appearance of the multispectral modalities. To effectively memorize contexts of the multispectral modalities, we propose a multispectral recalling loss. • Our pedestrian detector shows comparable performance to the multispectral pedestrian detector while receiving a single modal input regardless of modality type.

Related Work Pedestrian Detection in Single Modality
Pedestrian detection is one of the most widely studied topics in computer vision. It is applied to many human-related works, such as autonomous driving ) and video surveillance (Zhu and Peng 2015). From various previous hand-craft methods (Dalal and Triggs 2005;Benenson et al. 2014) to deep learning methods Wu et al. 2020b), a visible modality (e.g., RGB) is usually adopted, because this modality is the most intuitive modality for humans. Recent visible-based pedestrian detectors have introduced to alleviate the various issues, such as small-scale pedestrian Han et al. 2019;Wu et al. 2020b) and occlusion (Song et al. 2020;Xie et al. 2020b,a). Recently, some pedestrian detectors have been adopted a thermal modality, because this modality is robust to weather conditions (Kieu et al. 2019), low illumination (Zhou, Chen, and Cao 2020), and background clutter (Song et al. 2019). Several single modal pedestrian detectors use both visible modality and thermal modality in the training phase to improve the detection results. Xu et al. (Xu et al. 2017) proposed sub-networks that can improve the detection results of the visible modality using the information of the thermal modality. Kieu et al. (Kieu et al. 2019) utilized the visible modality of ImageNet (Krizhevsky, Sutskever, and Hinton 2012) and MS COCO (Lin et al. 2014) datasets prehand when conducting the thermal-based pedestrian detection. The work in (Kieu et al. 2020) proposed an auxiliary day/night classifier for adapting visible modality to the thermal modality. However, they mainly focused on handling one specific modality in the inference phase. As a result, it is difficult to effectively cope with the case when the other modal input comes in. In contrast, our method is flexible to any single input modality. Therefore, our detector can show robust performance day and night regardless of the input modal type, similar to the operation of surveillance camera.

Multispectral Pedestrian Detection
Recently, multispectral pedestrian detection has been received increasing attention by using the visible modality and the thermal modality together, and it shows remarkable detection performance (Park, Kim, and Sohn 2018;Zhang et al. 2019;Zhou, Chen, and Cao 2020;Li et al. 2019b;. Zhang et al. (Zhang et al. 2019) proposed AR-CNN to address the alignment issue of the two modalities. To reduce the effect of the modality discrepancy issue, Zhou et al. (Zhou, Chen, and Cao 2020) introduced MBNet which guides two modal features to be similar and selects the features according to the illumination score.
However, to apply the multispectral pedestrian detector to the real-world applications, image pairs of the two modalities should have the same FoV and same frame rate to avoid the miscalibration problem (Zhang et al. 2019). Kim et al.  somewhat mitigated this problem by adopting an uncertainty, but it still struggled when the FoV is significantly different (e.g., pedestrians are observed in one modality but not in the other modality). Also, the computational cost is higher than that of the single modal detectors. In contrast, our method is free from the problem, since we utilize the single modality in the inference phase. Furthermore, it shows comparable performance with the multispectral pedestrian detectors, because the MSR Memory can recall the multispectral modal information.

Memory Network
Memory augmented neural networks have been introduced, and they are applied to the various research fields (Zhu and Yang 2020;Marchetti et al. 2020;Lee et al. 2021;VS et al. 2021;. For example, Lee et al. (Lee et al. 2021) proposed LMC-Memory to predict the future frames by recalling long-term motion contexts. Recent works have adopted the key-value memory structure (Marchetti et al. 2020). Given query features, key memory calculates the similarity to address the relevant information of value memory. In this paper, we propose MSR Memory with multispectral   Figure 1: Network configuration of the proposed pedestrian detector in the training phase. Dotted box indicates that this procedure is considered only in the training phase. indicates the concatenation operation. Note that, in the inference phase, our detector freely receives any type of the single modality (visible (blue path) or thermal (red path)).
recalling loss to recall the multispectral information. Thus, our method can perform robust detection like multispectral pedestrian detector, regardless of the input modality. Figure 1 shows the overall architecture of the proposed pedestrian detector in the training phase. A weight-sharing backbone network receives an image pair of visible modality I v and thermal modality I t to encode image features of the two modalities. The two image features are passed through the RPN  to estimate N candidate regions, called Region of Interests (ROIs). With N ROIs, ROI align is conducted  to estimate ROI features of the visible modality F v and thermal modality F t . After that, F v and F t are passed through the weight sharing Multispectral Recalling (MSR) Memory. Let F v M and F t M denote output features of the MSR Memory. F v M and F t M are concatenated with original ROI features F v and F t , followed by 1×1 convolution to generate the refined ROI features R v and R t . Finally, classification and localization are conducted through a weight-sharing head network. Note that, in the inference phase, our pedestrian detector can receive any single input modality (i.e., visible (blue path) or thermal (red path)).

Multisensory-Matching Contrastive Loss
For the same scene, captured images of the two modalities are from different spectral bands. Therefore, the appearance of the observed pedestrians can be different. For example, in the dark night, the visible modality may lack color information to be judged as a pedestrian, but in the thermal modality, pedestrians may be well represented through the heat information. Therefore, we propose a multisensory-matching contrastive loss to guide the feature representation of the two modal ROI features F v and F t which are encoded by the backbone network to be similar. At the same time, it guides the ROI features of the pedestrian class close to each other and the pedestrian class and the background to be apart by referring the counterpart modality. F , where w, h, c indicate width, height, and channel, respectively) are passed through the global average pooling layer ). Let N p denote the number of ROI features of the pedestrian class among the N ROI features. When α modality is given (α = {v, t}), multisensory-matching contrastive loss to the counterpart β modality is defined as: where τ is the temperature parameter to control the softness. L α→β makes the N p pedestrian ROI features of the α modality and N p pedestrian ROI features of the β modality similar and make the background ROI features dissimilar. Based on the L α→β , we introduce the multisensorymatching contrastive loss L mmc by combining the ways of the two modalities, which can be represented as follows: With the L mmc , the backbone network can encode modalityagnostic ROI features, regardless of the input modality.

MSR Memory
Then, the similarity betweenf α i and L slots of the key memory M k is calculated for obtaining the addressing vector A i = {A i1 , ..., A iL } ∈ R 1×L which determines how to read the relevant information of the value memory M v . The j-th element of the A i is calculated as follows: If M kj is highly correlated withf α i , A ij will be high. In the opposite case, A ij will be low. After obtaining the addressing vector A i , flattened i-th output feature maps of the MSR Memoryf α Mi ∈ R 1×whc is calculated as: Finally,f α Mi is reshaped to generate i-th output feature maps of the MSR Memory f α Mi ∈ R 1×w×h×c . Note that, f α Mi is concatenated with f α i , followed by 1×1 convolution to obtain the refined i-th ROI feature r α i ∈ R 1×w×h×c . The weight parameters of the MSR Memory (M k and M v ) are initialized with Kaiming initialization  and they are updated through the overall training loss.

Multispectral Recalling Loss
Given F v and F t , main purpose of the proposed MSR Memory is to guide to be more discriminative features. It is known that the multispectral modalities contain abundant visual information about the pedestrians (Hwang et al. 2015). Therefore, we propose a multispectral recalling loss to guide the MSR Memory to store the information about the multispectral modalities.
To this end, we first generate the reference feature which can play a role of the teacher. As shown in Figure 1 (dotted box), F v is concatenated with F t to perform 1×1 convolution to encode the multispectral ROI feature R m ={r m i } N i=1 , r m i ∈ R 1×w×h×c . Note that, the reference multispectral ROI feature R m is generated only in the training phase.
After generating R m , the multispectral recalling loss guides R v and R t to be similar with R m . The multispectral recalling loss consists of feature similarity embedding loss and feature relation guiding loss. First, to guarantee the feature representation of each R v and R t to be closed with R m , we introduce feature similarity embedding loss L f se as: Through the L f se , values of each ROI feature of R v and R t become similar with the corresponding ROI features of R m . Second, we also consider the relationship between N ROI features in every single modality. To this end, R v and R t are passed through the global average pooling layer, followed by ℓ 2 normalization to obtain ). In the visible modality, similarity relation between N features are measured C v = l v l v ⊤ ∈ R N ×N . Then, we normalize C v via the softmax along the row direction to obtain similarity relation matrix S v . The i-th row vector S v i denotes the probability of how the i-th visible ROI feature is similar to the N visible ROI features. Ideally, the similarity between pedestrian ROI features may be high, vice versa between the pedestrian and background ROI features. Similarly, we obtain the similarity relation matrix of the thermal modality S t and the multispectral modalities S m .
With the three measurements (i.e., S v , S t , and S m ), we use KL divergence D KL (·) to compare the probability distributions. Through the KL divergence, we propose feature relation guiding loss L f rg to guide the relationship between ROI features to be similar with S m , which is defined as: Finally, the multispectral recalling loss L msr is obtained by adding L f se and L f rg , which is expressed as follows: L msr = L f se + L f rg .
(8) The L msr guides each ROI feature as well as the relationship between N ROI features to follow those of the reference features R m . By doing so, the MSR Memory can memorize information about pedestrian to improve the pedestrian feature representations of the original ROI features (i.e., F v and F t ). Consequently, the proposed detector can perform robust pedestrian detection with any single modal inputs by recalling multispectral modal features.

Total Loss Function
The total loss function is represented as follows: L T otal = L OD + λ 1 L mmc + λ 2 L msr , L OD = L RP N + L cls + L loc , where L OD includes loss function of the RPN, classification, and localization of two-stage object detectors . λ 1 and λ 2 denote balancing hyper-parameters.  Table 1: Detection results (MR) on KAIST dataset for the single modal inputs (V : visible, T : thermal). We compare our method with the multispectral pedestrian detectors. It assumes the environments where only one modality is available in the inference (Kim et al. 2019). When the multispectral pedestrian detectors takes one modal images, the counterpart modal images are blackout (fill all zero values), following ).

Experiments Datasets and Evaluation Metric
To validate the proposed pedestrian detector, we use two public datasets: (1) KAIST Multispectral Pedestrian Detection Dataset (Hwang et al. 2015) (for simplicity, we refer to this as KAIST dataset) and (2) CVC-14 (González et al. 2016). The two datasets contain the pair of visible-thermal images/annotations to detect the pedestrians.

KAIST Dataset
The KAIST dataset is a large-scale dataset with well-aligned visible-thermal image pairs (Hwang et al. 2015). It contains 95,328 visible-thermal image pairs with 103,128 bounding-box annotations and 1,182 unique pedestrians. The visible-thermal image pairs of the KAIST dataset are taken in a driving environment with various scenes. It includes the day and night conditions. Image resolution is 512×640. Note that, in the inference phase, we use test set with 2,252 images (visible or thermal).

CVC-14
The visible-thermal image pairs of CVC-14 are also collected in a driving environment with various scenes (González et al. 2016), including day/night conditions. Image resolution is 471×640. Following , we use 7,085 image pairs in the training phase and 1,433 images (visible or thermal) in the inference phase.
Evaluation Metric Following (Hwang et al. 2015), we adopt miss rate (MR) averaged over the false positive per image (FPPI) with range of [10 −2 , 10 0 ]. The lower MR denotes the better detection performance. We evaluate the performance on 'All', 'Day', and 'Night' environments.
To evaluate the performance, we divide the input modality into three cases: (1) visible (day/night), (2) thermal (day/night), and (3) visible (day) and thermal (night). Note that, case (3) imitates a surveillance camera that can change the input modality depends on the day/night time (Wu et al. 2020a). We call case (3) as 'mixed modality'.

Implementation Details
We implement the proposed detector based on Faster R-CNN  with VGG16 (Simonyan and Zisserman 2014) backbone. In ablation studies, we extend the baseline detector to Feature Pyramid Network (FPN) (Lin    (He et al. 2016) backbone to see the generalization ability of the proposed method. We train the proposed detector with stochastic gradient descent (SGD) which is synchronized over 4 GTX 1080 Ti GPUs with 4 images per mini-batch (1 image per GPU). We train our detector for 4 epochs with 0.008 learning rate. All experiments are conducted based on the Pytorch (Paszke et al. 2017). The number of ROIs is N = 256. We use the key-value slots number of the MSR Memory L = 50 as the default. Also, we use τ = 1 and λ 1 , λ 2 = 1.

Results on KAIST Dataset
First, we compare the proposed method with the state-ofthe-art multispectral pedestrian detectors (Li et al. 2018;Zhang et al. 2019;Zhou, Chen, and Cao 2020;. In the inference phase, a  Table 4: Effect of the proposed loss (i.e., L mmc and L msr ) on KAIST dataset (V : visible and T : thermal). 'Baseline (k)' indicates k modality is adopted to train the single-modal pedestrian detector in the training phase.  single modality is adopted. It assumes a real-world scenario where both modalities cannot be obtained at the same time (Kim et al. 2019). Therefore, when the multispectral pedestrian detectors take one modal inputs, we follow the settings of  where the counterpart modal inputs are blackout (fill all zero values). The results are shown in Table  1. Our method mostly surpasses the multispectral pedestrian detectors. The results show that multispectral pedestrian detectors may not flexibly handle a single modal input, while the proposed method can cope with any input modality. Next, we also compare our method with the state-of-theart thermal-based pedestrian detectors (Herrmann, Ruf, and Beyerer 2018;Guo, Huynh, and Solh 2019;Kieu et al. 2019;Zheng, Izzat, and Ziaee 2019;Kieu et al. 2020Kieu et al. , 2021Kim, Park, and Ro 2021a). Note that all the methods adopt images of the visible modality and the thermal modality in the training phase. As shown in Table 2, our method outperforms the existing thermal-based methods.

Results on CVC-14
We also conduct experiments on CVC-14. It is shown in Table 3. The 'Baseline (V + T )' adopting the two modalities in the training phase is better than 'Baseline (V )' and 'Baseline (T )'. In contrast, our method, which considers the multisensory-matching as well as the MSR Memory, outperforms all the baselines with large margin.

Ablation Studies
We conduct various ablation studies to investigate (1) effect of the proposed losses (i.e., L mmc and L msr ), (2) effect of the slot number L in MSR Memory, and (3)

Effect of the Proposed Losses
We evaluate the effectiveness of the proposed loss (multisensory-matching contrastive loss and multispectral recalling loss) by changing three types of input modality (i.e., visible, thermal, and 'mixed modality'). The results are shown in Table 4. When all the proposed losses are considered, the proposed method shows the more improved performance, compared to the baseline single-modal pedestrian detectors.

Slot Number of MSR Memory
We also conduct the experiments by modifying the slot number L of the MSR Memory as L = {25, 50, 100, 200}. Note that, we adopt the 'mixed modality' in the inference phase. As shown in Table  5, even we change L of the MSR Memory, performances of our method are still high, compared to the single-modal baseline pedestrian detector.
Baseline Detector Extension To see the generalization ability of our method with the MSR Memory, we adopt Feature Pyramid Network (FPN)  with ResNet-50 and -101 backbone (He et al. 2016). The input modality is the 'mixed-modality'. The results are shown in Table 6. Even we change the baseline detector, the proposed method with MSR Memory still outperforms the baselines.

Visualization Results
Following (Xu et al. 2020;Kim, Park, and Ro 2020), we conduct feature map visualization on KAIST dataset. Figure  3 shows some examples of feature map visualization results.  Figure 3: Feature map visualization ((a): thermal, (b): visible) of the original ROI feature, output feature of the MSR Memory, and refined ROI feature, following (Xu et al. 2020). Although the original ROI feature cannot focus the pedestrian regions, the refined ROI feature activate the proper regions (e.g., body) of the pedestrians with the help of the MSR Memory.
the MSR Memory recalls the information of the multispectral modalities to enhance the visual appearance of the ROI features by focusing on the pedestrian regions (e.g., body). For example, in the thermal modality of the daytime (see the top left of Figure 3), although F t mainly looks the pedestrian head region, R t can focus whole region of the pedestrian through the output feature of the MPR Memory F t M .

Discussions
Advantage of the Proposed Method According to Zhang et al. (Zhang et al. 2019), CVC-14 testset has a severe misalignment problem, where the observed pedestrian locations of the two modalities are significantly different because of the different FoVs. Therefore, the multispectral pedestrian detectors are hard to handle this problem. To mitigate this problem, AR-CNN (Zhang et al. 2019) and  were introduced. Thus, as shown in Table 7, they show the improved performance. However, since the proposed method uses a single modality but can act as a multispectral-like detector, so our method is free from the misalignment problem in the inference phase. As a result, our method mostly surpasses the state-of-the-art multispectral pedestrian detectors (Choi et al. 2016;Park, Kim, and Sohn 2018;Zhang et al. 2019;Zhou, Chen, and Cao 2020;).
Computational Cost We compare training/inference time and the network parameter number. It is shown in Table 8. Since the proposed method utilizes the visible modality as well as the thermal modality in the training phase, training time is similar to the baseline multispectral pedestrian detector (Faster R-CNN with VGG16). However, in the inference time, our method adopts a single modal input. Therefore, inference time and network parameters are similar to those of the baseline single-modal pedestrian detector.

Conclusion
In this paper, we propose a new perspective for pedestrian detection to effectively handle any single modal input. Here,   we design a multisensory-matching contrastive loss to make the pedestrian visual representation of two different modalities similar. Moreover, we devise MSR Memory with multispectral recalling loss to enhance the visual representations of the single modality by recalling appearance of the multispectral modalities. It enables the proposed pedestrian detector robust in terms of improving feature representation of the pedestrians observed in any single modality. Extensive quantitative and qualitative results verify the effectiveness of the proposed method. Although this method is developed for pedestrian detection task, we believe our method can provide insight into the studies dealing with various input modalities.