{'fraction_non_alphanumeric': 0.04019674966667868, 'fraction_numerical': 0.01500846816330943, 'mean_word_length': 4.988670694864048, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 5, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Pixel-wise semantic segmentation of RGB images can be advanced by exploiting informative features from supplementary modalities. In this work, we propose CMX, a vision-transformer-based cross-modal fusion framework for RGB-X semantic segmentation. To generalize to different sensing modalities encompassing various supplements and uncertainties, we consider that comprehensive cross-modal interactions should be provided. CMX is built with two streams to extract features from RGB images and the complementary modality (X-modality). In each feature extraction stage, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate the feature of the current modality by combining the feature from the other modality, in spatial- and channel-wise dimensions. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to mix them for the final semantic prediction. FFM is constructed with a cross-attention mechanism, which enables exchange of long-range contexts, enhancing both modalities' features at a global level. Extensive experiments show that CMX generalizes to diverse multi-modal combinations, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal and RGB-Polarization datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. Code is available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation.


INTRODUCTION
S EMANTIC segmentation is an essential task in computer vision, which aims to transform an image input into its underlying semantically meaningful regions and enables a pixelwise dense scene understanding for many real-world applications such as automated vehicles, robotics navigation, and augmented reality [1], [2], [3]. Over the last years, pixel-wise semantic segmentation of RGB images has gained an increasing amount of attention and made significant progress on segmentation accuracy [4], [5], [6]. Yet, due to the characteristics of RGB images, current deep semantic segmentation models cannot always extract high-quality features under some certain circumstances, e.g., when two objects have similar colors or textures, it is difficult to distinguish them through pure RGB images [7].
However, different types of sensors can supply RGB images with rich complementary information (see Fig. 1). For example, depth measurement can help identify the boundaries of objects and offer geometric information of dense scene elements [7], [8]. Thermal images facilitate to discern different objects through their specific infrared imaging [9], [10]. Besides, polarimetric-and event information are advantageous for perception in specularand dynamic real-world scenes [11], [12]. Therefore, multi-modal semantic segmentation models, that can effectively exploit such supplementary information, are often promising to yield better performances than single-modal RGB-based segmentation [13].
Existing multi-modal semantic segmentation methods can be divided into two categories. The first category employs a single network to extract features from RGB images, meanwhile information derived from another specific modality assist the extraction of RGB features ( Fig. 2(a)). This is a typical strategy in RGB-D semantic segmentation with depth-guided operators [14], [15], [16], which are usually elaborately tailored for specific modalities (e.g., depth), yet hard to be extended to operate with other modality combinations. The second type of approaches [7], [8], [10], [17] deploys two backbones to perform feature extraction from RGBand another modality separately, then fuses the extracted two features into one feature for semantic prediction (Fig. 2(b)). This arXiv:2203.04838v2 [cs.CV] 12 Apr 2022 structure offers the extensibility to replace the other modal-stream with alternative complementary modalities. Nevertheless, most previous methods are predominantly crafted for specific modalities, e.g., they only focus on tackling RGB-D scene parsing [7], [8], [18] or RGB-T segmentation [10], [17], [19]. While inspiring, they fail to work well on different multi-modal scenarios. For example, according to [10], [19], ACNet [7] and SA-Gate [8], designed for RGB-D data, perform less satisfactorily in RGB-T tasks (see Fig. 3). To enable robust real-world scene understanding, a general and flexible architecture suitable for a diverse mix of sensor data combinations enhancing RGB segmentation (Fig. 1), i.e. RGB-X semantic segmentation, is desirable and advantageous. It not only saves research and engineering efforts on optimizing architectures for a specific modality combination scenario, but also makes it possible that a system equipped with multi-modal sensors can readily leverage new sensors when they become available [20], [21], which is conductive to robust scene perception. However, such universal modality-agnostic RGB-X semantic segmentation architectures are rarely investigated in the state of the art.
Recently, with the emergence of powerful non-local feature extraction backbones like vision transformers [22], [23], [24], the performances of different dense prediction tasks, e.g., semantic segmentation, have been significantly advanced. Multi-modal data contain patterns that require modeling dependencies across modalities. Transformers handle inputs as sequences and have the ability to acquire long-range correlations [22], [25], offering the possibility for a unified framework for diverse multi-modal tasks. However, existing multi-modal fusion modules [7], [11], [17] are mostly architected for Convolutional Neural Networks (CNNs) that fail in establishing long-range contextual dependencies in pixel-rich image data, which prove essential for accurate semantic segmentation [26], [27]. Moreover, multi-modal data often encompass a great deal of noisy measurements in different sensing modalities, e.g., low-quality distance estimation regions caused by limited effective depth ranges [8] and uncertainties resulted from various event representations [12]. It remains unclear whether potential improvements on RGB-X semantic segmentation can be materialized via vision transformers. Crucially, while some previous works [7], [8] use a simple global multi-modal interaction strategy, it does not generalize well across different sensing data combinations [10]. We hypothesize that for RGB-X semantic segmentation with various supplements and uncertainties, comprehensive cross-modal interactions should be provided, to fully exploit the potential of cross-modal complementary features.
In light of these challenges, we propose CMX, a cross-modal fusion framework for RGB-X semantic segmentation. CMX uses RGB image and another modal image as inputs, with the aim of achieving better segmentation than using only RGB image. To extract bi-modal features, our vision-transformer-based model is built as a two-stream architecture (RGB-and X-modal stream). Two specific modules are designed for feature interaction and feature fusion in between. The first module, Cross-Modal Feature Rectification Module (CM-FRM), is put forward to calibrate the bi-modal features by leveraging their spatial-and channel-wise correlations, which enables both streams to focus more on the complementary informative cues from each other and mitigates the effects of uncertainties and noisy measurements from different modalities. Such a feature rectification tackles varying noises and uncertainties in diverse modalities. It enables better multi-modal feature extraction and interaction. The second module, Feature Fusion Module (FFM), is constructed in two stages, which merges both rectified features from RGB-and X-modality into a single feature for the semantic prediction. Motivated by the large receptive fields obtained via self-attention [25] of vision transformers, a cross-attention mechanism is devised in the first stage of FFM for realizing a cross-modal global reasoning. In the second stage, a mixed channel embedding is applied to produce enhanced output features. Thereby, our introduced comprehensive interactions lie in multiple levels (see Fig. 2(c)), including channel-and spatialwise rectification from the feature map perspective, as well as cross-attention from the sequence-to-sequence perspective, which are critical for generalization across modality combinations.
To verify our unification proposal, for the first time, we consider and assess CMX in four multi-modal semantic segmentation tasks. The proposed method achieves state-of-the-art performances on multiple RGB-Depth datasets [28], [29], [30], [31], [32], as well as RGB-Thermal [9] and RGB-Polarization [11] benchmarks. In particular, CMX attains top mIoU of 56.9% on NYU Depth V2 (RGB-D) [28], 59.7% on MFNet (RGB-T) [9], and 92.6% on ZJU-RGB-P [11] datasets. Our universal approach clearly outperforms specialized architectures (Fig. 3). Further, on popular challenging benchmarks [9], [28], CMX leads to ∼5.0% mIoU gains compared to RGB-only segmentation. Aside from these combinations of dense image data, we study dense-sparse fusion using dense RGB-and sparse event-based data (see Fig. 1), which encodes changes of intensity of each pixel asynchronously and thereby is constructive to capturing motion information [12]. However, an RGB-Event parsing benchmark is not available in the community. Tackling this shortage, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset [33], where our RGB-X model further shines, setting the new state-of-the-art among >10 benchmarked models. Meanwhile, the proposed CMX solution is proved to be effective for different CNN-and transformer backbone architectures. Moreover, our comprehensive set of investigations on representations of polarization-and event-based data indicates the path to follow and the sweet spot for reaching robust multi-modal semantic segmentation, trumping original representation methods [11], [33].
On a glance, we deliver the following contributions: • For the first time, explore RGB-X semantic segmentation in four multi-modal sensing data combinations. • Rethink cross-modal fusion from a generalization perspective and consider that comprehensive cross-modal interactions should be provided to generalize across diverse modalities. • Propose an RGB-X semantic segmentation framework CMX with cross-modal feature rectification and feature fusion modules, intertwining cross-attention and mixed channel embedding for enhanced global reasoning with transformers. • Investigate representations of polarimetric-and event data, indicating the optimal path to follow for reaching robust multi-modal semantic segmentation. • Establish an RGB-Event semantic segmentation benchmark to assess dense-sparse data fusion. Our general method achieves state-of-the-art performances on eight datasets, covering five RGB-D benchmarks, as well as RGB-T, RGB-P, and RGB-E sensing data combinations.

Transformer-driven Semantic Segmentation
Since Fully Convolutional Networks (FCNs) [34] address dense semantic segmentation in an end-to-end per-pixel classification a) Input Fusion b) Feature Fusion c) Interactive Fusion feature-wise feature-wise sequence-wise Fig. 2: Comparison of different fusion methods. a) Input fusion merges the input with modality-specific operations [15], [16]. b) Feature fusion uses a global feature-wise strategy via channel attention [7], [8]. c) In our cross-modal fusion framework for RGB-X semantic segmentation with transformers, comprehensive interactions are considered and provided, including channel-and spatial-wise cross-modal feature rectification from the feature map perspective, as well as cross-attention from the sequence-to-sequence perspective.  [28]), is less effective on RGB-T or RGB-E tasks. CMX, for the first time, outperforms state-of-the-art specialized methods on four studied segmentation tasks on multiple datasets.
For multi-modal semantic segmentation, there are two dominant strategies. The first mainstream paradigm models crossmodal complementary information into layer-or operator designs [14], [15], [16], [93], [94]. For example, a shape-aware convolution layer is introduced in [15] for embedding depth features, while a spatial-guided convolution operator is proposed in [16] to adapt receptive fields according to geometric patterns for RGB-D semantic segmentation. Additional methods dynamically exchange channels and shift pixels to enable implicit cross-modal feature fusion [95], [96]. While these works verify that multimodal features can be learned within a shared network, they are carefully designed for some specific modality combinations, in particular RGB-D semantic segmentation, which are hard to be applied into other modalities. Moreover, there are multi-task frameworks [97], [98], [99], [100], [101] that facilitate inter-task feature propagation for RGB-D scene understanding, but they rely on supervision from other tasks for joint learning.
The second paradigm dedicates to developing fusion schemes to bridge two parallel modality streams. ACNet [7] proposes attention complementary modules to exploit high-quality informative features for RGB-D semantic segmentation based on channel attention [102], whereas ABMDRNet [10] suggests to reduce the modality differences between RGB-and thermal features before selectively extracting discriminative cues for RGB-T fusion. For RGB-P segmentation, Xiang et al. [11] connect RGB-and polarization branches via efficient channel attention bridges. For RGB-E parsing, Zhang et al. [12] explore sparse-to-dense and denseto-sparse fusion flows, verifying the importance of event-driven dynamic context for accident scene segmentation. In this research, we also advocate this paradigm but unlike previous works, we address RGB-X semantic segmentation with a unified framework, for generalizing to diverse sensing modality combinations.
While some previous works use a simple global channelwise interaction strategy, it does not work well across different sensing data. For example, ACNet [7] and SA-Gate [8], designed for RGB-D segmentation, perform less satisfactorily in RGB-T scene parsing [10]. In contrast, we hypothesize that comprehensive cross-modal interactions should be provided for RGB-X semantic segmentation with various supplements and uncertainties, to fully unleash the potential of cross-modal complementary features. Besides, most of the previous works still adopt CNNs as the backbone without considering that long-range dependency information could be informative for cross-modal fusion. We put forward a framework with transformers which considers global dependencies already in its architecture design. Concurrent works also use transformers for multi-modal action recognition [103], object detection [104], [105], [106], and end-to-end self-driving [107]. Differing from existing works, we perform fusion on different levels with cross-modal feature rectification and cross-attentional exchanging for enhanced dense semantic prediction.

PROPOSED FRAMEWORK: CMX
We first introduce the overview of our proposed CMX framework for RGB-X semantic segmentation in Sec. 3.1. In Sec. 3.2 and Sec. 3.3, we elaborate the proposed CM-FFM and FFM modules for cross-modal feature rectification and fusion. In Sec. 3.4, we detail the used multi-modal data representations for investigating and reaching robust RGB-X semantic segmentation.

Framework Overview
We present CMX, a cross-modal fusion framework for RGB-X semantic segmentation. The overview of CMX is shown in Fig. 4. We use two parallel backbones to extract features from RGB-and X-modal inputs, which can be RGB-depth, -thermal, -polarization, -event data, etc. Multi-modal data often contain a great amount of noises in different modalities, e.g., low-quality depth regions caused by limited effective distance measuring ranges [8] and uncertainties resulted from various event representations [12].
While features from different modalities have their specific noisy measurements, the feature of another modality has the potential for rectifying and calibrating this noisy information. Therefore, we design a Cross-Modal Feature Rectification Module (CM-FRM) to rectify features coming from both modalities. They will be assembled between two adjacent stages of backbones. In this way, both rectified features will be sent to the next stage to further deepen and improve the feature extraction. Furthermore, we design a two-stage Feature Fusion Module (FFM) to fuse features belonging to the same level into a single feature map. Then, a decoder is used to transform the feature maps of different levels into the final semantic map. With the aim to harvest diverse supplements and tackle varying uncertainties, our introduced comprehensive interactions lie in multiple levels, including channeland spatial-wise rectification from the feature map perspective in CM-FRM, and cross-attention from the sequence-to-sequence perspective in FFM, which are crucial for generalizing across sensing data combinations and reaching robust RGB-X semantic segmentation. The backbones and the decoder can be replaced by other common methods. In Sec. 3.2 and Sec. 3.3, we detail the design of CM-FRM and FFM, respectively. In the following descriptions, we use X to refer to the supplementary modality, which can be depth-, thermal-, polarization-, event data, etc.

Cross-Modal Feature Rectification
As analyzed above, the information originating from different sensing modalities are usually complementary [7], [8] but contain noisy measurements. The noisy information can be filtered and calibrated by using features coming from another modality. To this purpose, we propose a novel Cross-Modal Feature Rectification Module (CM-FRM) to perform feature rectification between parallel streams at each stage during feature extraction. For tackling various noises and uncertainties in diverse modalities, CM-FRM processes the input features in two different dimensions, including channel-wise and spatial-wise feature rectification, which together offers a holistic calibration, enabling better multi-modal feature extraction and interaction. Channel-wise feature rectification. We embed both modalities' features RGB in ∈ R H×W×C and X in ∈ R H×W×C along the spatial axis into two attention vectors W C RGB ∈ R C and W C X ∈ R C . Different from previous channel-wise attention methods [8], [17], [108], which use global average pooling to obtain the attention vector, we apply both global max pooling and global average pooling to RGB in and X in along the channel dimension to remain more information. We concatenate the four resulted vectors, having Y ∈ R 4C . Then, an MLP is applied, followed with a sigmoid function to obtain the weights W C ∈ R 2C from Y, which will be split into W C RGB and W C X : where σ denotes the sigmoid function. The channel-wise rectification is then operated as: where denotes channel-wise multiplication. Spatial-wise feature rectification. As the aforementioned channel-wise feature rectification module concentrates on learning global weights for a global calibration, we further introduce a spatial-wise feature rectification for calibrating local information. The two modalities' inputs RGB in and X in will be concatenated and embedded into two spatial weight maps: W S RGB ∈ R H×W and W S X ∈ R H×W . The embedding operation has two 1×1 convolution layers assembled with a RELU function. Afterwards, a sigmoid function is applied to obtain the embedded feature map F ∈ R H×W×2 , which is further split into two weight maps. The process to obtain the spatial weight maps is formulated as: Similar to the channel-wise rectification, the spatial-wise rectification is formulated as: where * denotes spatial-wise multiplication. The whole rectified feature for both modalities RGB out and X out is organized as: λ C and λ S are two hyperparameters and we set them both as 0.5. RGB out and X out are the rectified features after the comprehensive calibration, which will be sent into the next stage for feature fusion.

Feature Fusion
After obtaining the feature maps at each layer, we build a twostage Feature Fusion Module (FFM) to enhance the information interaction and merge the features from two modalities into a single feature map. As shown in Fig. 4(c), in stage 1, the two branches are still maintained, and a cross-attention mechanism is designed to globally exchange information between the two branches. Then, we concatenate the outputs of these two branches.
In stage 2, this concatenated feature is transformed into the original size via a mixed channel embedding. Information exchange stage. At this stage, the two modalities' features will exchange their information via a symmetric dual-path structure. For brevity, we take the X-modal path for illustration. We first flatten the input feature with size R H×W ×C to R N ×C , where N =H×W . Afterwards, a linear embedding is used to generate two vectors with the same size R N ×Ci , which we call residual vector X res and interactive vector X inter . We further put forward an efficient cross-attention mechanism applied to these two interactive vectors from different modal paths, which will carry out sufficient information exchange across modalities. This offers complementary interactions from the sequence-to-sequence perspective beyond the rectification-based interactions from the feature map perspective in CM-FRM. Our cross-attention mechanism for enhancing cross-modal feature fusion is based on the traditional self-attention [25]. The original self-attention operation encodes the input vectors into Query Q, Key K, and Value V. The global attention map is calculated via a matrix multiplication QK T , which has a size of R N ×N and causes a high memory occupation. In contrast, [109] uses a global context vector G = K T V with a size R C head ×C head and the attention result is calculated by QG. We flexibly adopt the reformulation and develop our multi-head cross-attention based on this efficient self-attention mechanism. Specifically, the interactive vectors will be embedded into Key K and Value V for each head, and both sizes of them are R N ×C head . The output is obtained by multiplying the interactive vector and the context vector from the other modality path, namely a crossattention process, and it is depicted in the following equations: Note that G denotes the global context vector, while U indicates the attended result. To realize the attention from different representation subspaces, we remain the multi-head mechanism, where the number of heads matches the transformer backbone. Then, the attended result vector U and the residual vector X res are concatenated. Finally, we apply a second linear embedding and resize the feature to R H×W ×C . Fusion stage. In the second stage of FFM, precisely the fusion stage, we use a simple channel embedding to merge the two paths' features, which is realized via 1×1 convolution layers. Further, we consider that during such a channel-wise fusion, the information of surrounding areas should also be exploited for robust RGB-X segmentation. Thereby, inspired by Mix-FFN in [61] and ConvMLP [110], we add one more depth-wise convolution layer DW Conv 3×3 to realize a skip-connected structure. In this way, the merged features with the size R H×W ×2C are fused into the final output with the size of R H×W ×C for feature decoding.

Multi-modal Data Representations
To investigate and reach robust RGB-X semantic segmentation, the multi-modal data representations are critical. We study with four modality combinations, including RGB-Depth, RGB-Thermal, RGB-Polarization, and RGB-Event semantic segmentation. Now we elaborate the multi-modal data representations we use for inputting to our symmetric dual-path CMX architecture. RGB-Depth. Depth images naturally offer range, position, and contour information. The fusion of RGB information and depth information is promising to better separate objects with indistinguishable colors and textures at different spatial locations. We encode the raw depth images into HHA images as the depth modality input. This algorithm is proposed by [111], which has been proven to have a better use of depth information. Precisely, HHA offers a representation of geometric properties at each pixel, including horizontal disparity, height above ground, and the angle the pixel's local surface makes with the inferred gravity direction, emphasizing complementary discontinuities of RGB images. RGB-Thermal. At night or in places with insufficient light, objects and backgrounds have similar color information and are difficult to distinguish. Thermal images provide infrared characteristics of objects, which are potential to improve objects with thermal properties such as people. We directly use the infrared thermal image and copy the single-channel thermal image input 3 times to match the backbone input. RGB-Polarization. High-reflectivity objects such as glasses and cars have RGB information that are easily confused with surroundings. Polarization cameras record the optical polarimetric information when polarized reflection occurs, which offer complementary information serviceable for better segmenting and understanding scenes with specular surfaces. The polarization sensor equipped with a polarization mask layer with four different polarization directions [11] and thereby each captured image set consists of four pixel-aligned images at different polarization angles where I angle denotes the image recorded at the corresponding angle. We investigate representations of polarization data, including the Degree of Linear Polarization (DoLP ) and the Angle of Linear Polarization (AoLP ), which are key polarimetric properties characterizing light polarization patterns [11]. They are derived by Stokes vectors S = [S 0 , S 1 , S 2 , S 3 ] that describe the polarization state of light. Precisely, S 0 represents the total light intensity, S 1 and S 2 denote the ratio of 0 • and 45 • linear polarization over its perpendicular polarized portion, and S 3 stands for the circular polarization power which is not involved in our work. The Stokes vectors S 0 , S 1 , S 2 can be calculated from image intensity measurements Then, DoLP and AoLP are formally computed as: In our experiments, we further study with monochromatic and trichromatic polarization cues, coupled with RGB images in multi-modal RGB-P semantic segmentation. For monochromatic representation used in previous works [11], [112], we obtain it from monochromatic intensity measurements and convert it to 3-channel input by copying the single-channel information. For trichromatic polarization representation in either DoLP or AoLP , we compute separately for their respective RGB channels. RGB-Event. Event data provide multiple advantages such high dynamic range, high temporal resolution, and not being influenced by motion blur [89], which are critical in dynamic scenes with a great amount of motion information such as road-driving environments [12], [90]. Due to the sparse nature of event data, the raw events require a pre-processing to obtain a representation to achieve multi-modal fusion. Events are typically converted into a tensor with the same spatial dimension as the corresponding RGB image. For example, a set of raw events in a time window ∆T =t N −t 1 is embedded into a voxel grid with spatial dimensions H×W and time bins B, where t 1 and t N are the startand the end time stamp of the raw events. For example, event data in [33] is directly converted to B=3. In this work, events are first embedded into a voxel grid with a higher time resolution, which we set the upscale size of the event bin as 6. Then, every 6 adjacent panels are superimposed to obtain a tensor with fine-grained event embedding. A comparison between the direct representation in [33] and our event representation is visualized in Fig. 5, in which our event representation is more fine-grained in each event panel. Apart from B=3, we further investigate different settings of event time bin B={1, 5, 10, 15, 20, 30} in our method for reaching robust RGB-E semantic segmentation.

EXPERIMENT DATASETS AND SETUPS
In this section, we describe the multi-modal datasets and implementation details of our model in experiments.

Datasets
We use multiple RGB-Depth semantic segmentation datases, as well as datasets of RGB-Thermal, RGB-Polarization, RGB-Event modality combinations for verifying our proposed CMX model. RGB-E EventScape dataset. While event-based data offer various benefits for perception in dynamic scenes [12], [90], a largescale multi-modal RGB-Event semantic segmentation benchmark is not available in the field. Addressing this gap, we create an RGB-E multi-modal semantic segmentation benchmark 1 based on the EventScape dataset [33], which is originally designed for depth estimation. To maintain data diversity from the video sequences of the EventScape dataset generated from the CARLA simulator [114], we select one frame from every 30 frames. Thus, there are 4077 and 749 images in the training-and evaluation set, respectively. The images have a 512×256 resolution and are annotated with 12 semantic classes.

Implementation Details
During training on all datasets, data augmentation is performed by random flipping and scaling with random scales [0.5, 1.75].
We take Mix Transformer encoder (MiT) pretrained on Ima-geNet [115] as the backbone and MLP-decoder with an embedding dimension of 512 unless specified, both introduced in Seg-Former [61]. We select AdamW optimizer [116] with weight decay 0.01. The original learning rate is set as 6e −5 and we employ a poly learning rate schedule. We use cross-entropy as the loss function. When reporting multi-scale testing results on NYU Depth V2 and SUN RGB-D, we use multiple scales ({0.75, 1, 1.25}) with horizontal flipping. We use mean Intersection over Union (mIoU) averaged across semantic classes as the primary evaluation metric to measure the segmentation performance. More specific settings for different datasets are described in detail in the appendix.

EXPERIMENT RESULTS AND ANALYSES
In this section, we present experimental results for verifying the effectiveness of our proposed CMX approach for RGB-X semantic segmentation. In Sec. 5.1, we show the performances of our model on multiple indoor and outdoor RGB-Depth benchmarks, compared with state-of-the-art methods. In Sec. 5.2, we analyze 1. https://paperswithcode.com/sota/semantic-segmentation-on-eventscape  the RGB-Thermal segmentation performance for robust daytimeand nighttime semantic perception. In Sec. 5.3 and Sec. 5.4, we study the generalization of CMX to RGB-Polarization and RGB-Event modality combinations and representations of these multi-modal data. We conduct a comprehensive variety of ablation studies in Sec. 5.5 to confirm the effects of different components in our solution. Finally, we perform efficiency-and qualitative analysis in Sec. 5.6 and Sec. 5.7.

Results on RGB-Depth Datasets
We first assess the performance of our method on different RGB-D semantic segmentation datasets. NYU Depth V2. The results on the NYU Depth V2 dataset are shown in Table 1. The performances of our model using different sizes of backbones are reported. It can be easily seen that our approach achieves leading scores. The proposed method with SegFormer-B2 already exceeds previous methods, attaining 54.4% in mIoU. Our CMX models based on SegFormer-B4 and -B5 further dramatically improve the mIoU to 56.3% and 56.9%, clearly standing out in front of all state-of-the-art approaches. The best CMX model even reaches superior results than recent strong pretraining-based methods [21], [101] like Omnivore [21] that uses images, videos, and single-view 3D data for supervision. SUN-RGBD. The segmentation results of CMX on the SUN-RGBD dataset are shown in Table 2. We experiment by training and testing backbones of two different sizes and our method achieves leading performances. Our interactive cross-modal fusion   approach exceeds previous works that either use input fusion depicted in Fig. 2(a) including SGNet [16] and ShapeConv [15], or build on feature fusion illustrated in Fig. 2(b) including ACNet [7] and SA-Gate [8]. In particular, with SegFormer-B4 and -B5, our proposed method elevates the state-of-the-art to more than 52.0% in mIoU. CMX is also better than multi-task methods like PAP [97] and TET [100]. As the model with SegFormer-B5 comes with much larger computational complexity and relatively small improvements, which will be assessed in Sec. 5.6, in the following experiments we mainly use SegFormer-B2 and -B4 backbones.
Stanford2D3D. As shown in Table 3, on the large-scale Stan-ford2D3D dataset, our proposed solution also achieves stateof-the-art mIoU scores. CMX based on SegFormer-B2 already surpasses the previous best ShapeConv [15] based on ResNet-101 [124] in mIoU and our model based on SegFormer-B4 further improves the accuracy values, reaching mIoU of 62.1%. The results demonstrate the effectiveness and learning capacity of our approach on such a large RGB-D dataset. ScanNetV2. We have also tested our CMX model with SegFormer-B2 on the ScanNetV2 benchmark. As shown in Table 4, RGB-D methods generally obtain better performance than RGB-only methods. It can be clearly seen that CMX outperforms RGB methods and achieves the top mIoU of 61.3% among the RGB-D methods. On the ScanNetV2 leaderboard, while there are methods like BPNet [98] that reaches a higher score, they rely on 3D supervision from point clouds to perform joint 2D-and 3D reasoning. In contrast, our method attains a competitively accurate performance by using purely 2D data and effectively leveraging the complementary information inside RGB-D modalities.
Cityscapes. The previous results are reported on indoor RGB-D datasets. To study the generalizability to outdoor scenes, we assess the effectiveness of CMX on Cityscapes, a well-known semantic segmentation dataset of RGB-D road-driving environments. As shown in Table 5, we compare the performances of our models against state-of-the-art RGB and RGB-D methods. Compared with SegFormer-B2 (RGB), our RGB-D approach elevates the accuracy by 0.6% in mIoU. Our approach based on SegFormer-B4 achieves a state-of-the-art score of 82.6% among the competitive models, outstripping all existing RGB-D methods on Cityscapes by more than 0.4% in absolute mIoU values, verifying that CMX generalizes well to street scene understanding.

Results on RGB-Thermal Dataset
We then investigate the generalizability of our approach to another modality combination scenario, precisely, RGB-Thermal semantic segmentation on the MFNet dataset [9].
Comparison with the state-of-the-art. In Table 6, we compare our method against RGB-only models and multi-modal methods using RGB-T inputs. As unfolded, ACNet [7] and SA-Gate [8], designed for RGB-D segmentation, perform less satisfactorily on RGB-T data, as they only use a global channel-wise fusion strategy and thereby fail to fully exploit the complementary cues. Depth-aware CNN [14], an input fusion method with modalityspecific operator design, also does not yield high performance. In contrast, the proposed CMX strategy, enabling comprehensive interactions from various perspectives, generalizes smoothly in RGB-T semantic segmentation. It can be seen that our method based on SegFormer-B2 achieves mIoU of 58.2%, clearly outperforming the previous best RGB-T methods ABMDRNet [10], FEANet [17], and GMNet [19]. Our CMX solution based on SegFormer-B4 further elevates state-of-the-art mIoU to 59.7%, widening the accuracy gap in contrast to existing methods. Moreover, it is worth pointing out that the improvements brought by our RGB-X approach compared with the RGB baselines are compelling, i.e., +5.0% and +4.9% in mIoU for SegFormer-B2 and -B4 backbones, respectively. Our approach overall achieves top scores on car, person, bike, curve, car stop, and bump. For person with infrared properties, our approach enjoys more than +11.0% gain in IoU, confirming the effectiveness of CMX in harvesting complementary cross-modal information.
Day and night performances. Following [19], [84], we assess day-and night segmentation results of our models compared against existing methods. For daytime scenes, both RGB-and thermal images provide beneficial meaningful information, and our approach increases IoU by 2.7%∼3.1% compared with RGBonly baselines. At the nighttime, RGB segmentation often suffers from poor lighting conditions, and it even carries much noisy information in the RGB data. Yet, our CMX rectifies the noisy images and exploits supplementary features from thermal data, dramatically improving the performances by >7.0% in mIoU at night and enhancing the robustness of semantic scene understanding in unfavorable environments with adverse illuminations.

Results on RGB-Polarization Dataset
We now extend the study to RGB-P semantic segmentation on the ZJU-RGB-P dataset [11].
Comparison with the state-of-the-art. Table 8 shows per-class accuracy of our approach compared against RGB-only [61], [130] and RGB-Polarization fusion methods [11], [112]. Our approach largely elevates state-of-the-art performance, outperforming the previous best RGB-P method [11] by >6.0% in mIoU. We observe that the accuracy improvement on pedestrian is significant thanks to the capacity of the transformer backbone and our crossmodal fusion mechanisms. The IoU improvements compared to the results using only RGB modality (SegFormer-B2 [61]) are clear on classes with polarimetric characteristics such as glass (>8.0%) and car (>2.5%), further evidencing the generalizability of our cross-modal fusion solution in bridging RGB-P streams.
Polarization data representations. We study polarimetric data representations and the results displayed in Table 8 indicate that AoLP and DoLP representations both carry effective polarization information beneficial for semantic scene understanding, which is consistent with the finding in [11]. Besides, trichromatic representations are consistently better than monochromatic representations used in previous RGB-P segmentation works [11], [112]. This is expected as the trichromatic representation provides more detailed information, which should be leveraged to fully unlock the potential of trichromatic polarization cameras.

Results on RGB-Event Dataset
We take a step further to assess the generalizability of our proposed approach for dense-sparse data fusion.
Comparison with the state-of-the-art. In Table 9, we benchmark more than 10 semantic segmentation methods. We compare our models against RGB-only methods covering CNNbased SwiftNet [130], Fast-SCNN [141], CGNet [142], and DeepLabV3+ [143], as well as transformer-based Swin [24], SegFormer [61], and Trans4Trans [3]. We also include multimodal methods, spanning RFNet [1] designed for road-driving scene segmentation and ISSAFE [12], the only known RGB-Event method designed for traffic accident scene segmentation, as well as SA-Gate [8], a state-of-the-art RGB-D segmentation method. While efficiently attaining accuracy improvements over SwiftNet, RFNet and ISSAFE overall only achieve rather unsatisfactory mIoU scores. SA-Gate, crafted for RGB-D data, is also less effective in the RGB-E combination scenario. Compared with existing methods, our models improve segmentation performance by mixing RGB-Event features, as it can be clearly seen in Table 9 and Fig. 6. Our model using MiT-B4 as the backbone reaches 64.28% in mIoU, towering over all other methods and setting the state-of-the-art on the RGB-E benchmark. This further verifies the versatility of our solution for different multi-modal combinations. Moreover, we experiment with DeepLabV3+ [143] and Swin transformer with UperNet [144] in dual-branch structures by using their multi-stage features with CMX. The results show that our RGB-X solution consistently improves the scores of RGB-only models, confirming that our method is not strictly tied to a concrete backbone architecture, but can be easily deployed with other CNN-and transformer models, which helps to yield effective models for multi-modal semantic segmentation. Fig. 6 depicts per-class accuracy comparison between the RGB baseline and our RGB-Event model with   SegFormer-B2. With event data, the foreground objects are more accurately parsed by our RGB-E model, such as vehicle (+2.1%), pedestrian (+11.7%), and traffic light (+7.0%). Event data representations. In Table 10, we further study with different settings of event time bin B={1, 3, 5, 10, 15, 20, 30} with our CMX fusion model based on SegFormer B2. Compared with the original event representation [33], our method achieves consistent improvements (visualized in Fig. 7) on different settings of event time bins, such as +1.63% of mIoU when B=30. In particular, it helps our CMX to obtain the highest mIoU of 61.90% in the setting of B=3. In B=1, embedding all events in a single time bin leads to dragging behind images of moving objects and being sub-optimal for feature fusion. In higher time bins, events produced in a short interval are dispersed to more bins, resulting in insufficient events in a single bin. These corroborate relevant observations in [12], [90] and that the event representation B=3 is an effective time bin setting for RGB-E semantic segmentation with CMX. We note that the models in Table 10 are trained on an A100 GPU, and CMX produces a slightly different pixel accuracy score compared to that of ours (SegFormer-B2) in Table 9.

Ablation Study
We perform a series of ablation studies to explore how different parts of our architecture affect the segmentation. We use depth information encoded into HHA as the complementary modality here. We take MiT-B2 as the backbone with the MLP decoder in our ablation studies, unless specified. The semantic segmentation performance is evaluated on the NYU Depth V2 dataset.
Effectiveness of CM-FRM and FFM. We design CM-FRM and FFM to rectify and merge features coming from the RGBand X-modality branches. We take out these two modules from the architecture respectively, where the results are shown in Table 11. If CM-FRM is ablated, the features will be extracted independently in their own branches, and for FFM we simply average the two features for semantic prediction. Compared with the baseline, using only CM-FRM improves mIoU by 2.5%, using only FFM improves mIoU by 1.2%, and together CM-FRM and FFM improve the semantic segmentation performance by 3.8%. Ablation with CM-FRM and FFM variants. We further experiment with variants of the CM-FRM and FFM modules. As shown in Table 12, channel only denotes using channel-wise rectification only (λ C =0 and λ S =1), and spatial only means using spatialwise rectification only (λ C =0 and λ S =1). It can be seen that replacing the proposed CM-FRM by either channel only or spatial only causes a sub-optimal accuracy, further confirming the efficacy of combining the two dimensions' rectification for holistic feature calibration, which is crucial for robust multi-modal segmentation. In our channel-wise calibration, we use both global average pooling and global max pooling to remain more information. The results in Table 12 show the effects of using only global average pooling (avg. p.) and using only global max pooling (max. p.). Both of them are less effective than our complete CM-FRM, which offers a more comprehensive rectification. Previous ablation studies support the design of CM-FRM. To understand the capability of FFM, we here test with two variants. As shown in Table 12, stage 2 only means there is no information exchange before the mixed channel embedding, whereas self attn denotes that the context vectors will not be exchanged in the stage 1 of FFM. The proposed modules productively rectify and fuse the features in different levels. Specifically, the experiments with the FFM variants illustrate the effectiveness of our cross-attention These indicate the importance of fusion from the sequence-tosequence perspective, which is not considered in previous works.
Overall, the ablation shows that our interactive strategy, providing comprehensive interactions, is effective for cross-modal fusion.
Ablation of the supplementary modality. Previous works have shown that multi-modal segmentation has a better performance than single-modal RGB segmentation [7]. We carry out experiments to certify and the results are shown in Table 13. Note that here, the MLP decoder is not used, in order to focus on studying the influence of feature extraction from different supplementary modalities. First, we use only a single MiT-B2 backbone which uses only RGB information. Then, due to our approach using two parallel backbones, to facilitate a fair comparison we keep a dualpath architecture with both RGB inputs, which results slightly better than a single backbone. Additionally, we replace the second modality with random noise, which obtains even better results than two RGB inputs. This means that even pure noise information may help the model to identify noisy information in the RGB branch.
The model learns to focus on relevant features and thus gains robustness. It may also help prevent over-fitting during the learning process. However, when using depth information, we have observed obvious improvements compared to these three settings, which further proves that the fusion of RGB and depth information brings clearly better predictions. Encoding depth images using the HHA representation further increases the scores. The overall gain of 5.3% in mIoU, compared with the RGB-only baseline, is also compelling, which is similar to that in RGB-T semantic segmentation, demonstrating the effectiveness of our proposed method for rectifying and fusing cross-modal information.

Efficiency Analysis
In

Qualitative Analysis
In our final study, we showcase various examples of representative qualitative results to deepen the analysis. Segmentation results. We visualize four segmentation results across different modality combinations with the backbone of SegFormer-B2 in Fig. 8. The proposed method can harvest complementary information from the second modality and thereby yields more accuracy segmentation results. For RGB-D semantic segmentation in an indoor scene from the NYU Depth V2 dataset [28], CMX leverages geometric information and correctly identifies the bed while the RGB-only model wrongly classifies it as a sofa. For RGB-T segmentation, while the baseline yields very coarse segmentation under the low illumination conditions at night, much clearer boundary distinctions are made by our method between the persons and unlabeled background with infrared information acquired in the thermal modality and fused via our solution. For RGB-P segmentation, specular glass areas are more precisely parsed, the cars which also contain polarization cues are completely and smoothly segmented with delineated borders, and the pedestrian detection also shows beneficial effects. Moreover, for dense-sparse fusion of RGB-Event data, our method generalizes well and enhances the segmentation of moving objects. Overall, the qualitative examination backs up that our general approach is suitable for a diverse mix of multi-modal sensing combinations for robust semantic scene understanding. We further view the outdoor RGB-D semantic segmentation results on the Cityscapes dataset [32] based on the backbone of SegFormer-B4. We show the results of the baseline RGB-only model and our RGB-X approach, in particular the difference maps w.r.t. the segmentation ground truth. As displayed in Fig. 9, in spite of the noisy depth measurements, the HHA image encoding depth information still benefits the segmentation via our model that is able to rectify and fuse cross-modal complementary features. It can be seen that our approach has better pixel accuracy scores on a wide variety of driving scene elements such as fence, rider, sidewalk, and road in the positive group, which are highlighted in green boxes. However, the shadows and weak illumination conditions are still challenging for both models and make the depth cues less effective. For example, depth information in the regions of sidewalk in the negative group, may be less informative for fusion, as highlighted in the red boxes.
Feature analysis. To understand the key module for feature rectification, we visualize the input-and rectified features of CM-FRM in layer 1, and their difference map, as shown in Fig. 10. It can be seen that the feature maps are enhanced in both streams after the cross-modal calibration. The RGB-stream delivers texture information to the supplement modality, while the supplement modality further improves the boundary and emphasizes complementary discontinuities of RGB features. In the RGB-D segmentation scenario, the RGB-feature difference map shows that the ground area is better spotlighted, thanks to the HHA image encoding depth information, which provides geometric cues such as height above ground, beneficial for higher-level semantic prediction of groundrelated classes. In the RGB-T nighttime scene parsing cases, the pedestrians are hard to be seen in the RGB images. But the RGBfeature difference map clearly highlights the pedestrians thanks to the supplementary thermal modality with infrared imaging. These indicate that the complementary features have been infused into the RGB-stream. The RGB features have been rectified to better focus on informative ones and capture such complementary discontinuities towards accurate semantic understanding.

CONCLUSION
To revitalize multi-modal pixel-wise semantic scene understanding, we investigate RGB-X semantic segmentation and propose CMX, a universal vision-transformer-based cross-modal fusion architecture, which is generalizable to a diverse mix of sensing data combinations. We put forward a Cross-Modal Feature Rectification Module (CM-FRM) and a Feature Fusion Module (FFM) for facilitating interactions towards accurate RGB-X semantic segmentation. CM-FRM conducts channel-and spatialwise rectification, rendering comprehensive feature calibration. FFM intertwines cross-attention and mixed channel embedding for enhanced global information exchange. To further assess the generalizability of CMX to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark. We study effective representations of polarimetric-and event data, indicating the optimal path to follow for reaching robust multi-modal semantic segmentation. The proposed model sets the new state-of-the-art on eight benchmarks, spanning five RGB-D datasets, as well as RGB-Thermal, RGB-Polarization, and RGB-Event combinations.
In the future, we aim to extend CMX to a three-path architecture for combing three or more types of sensor data. Another possible research direction is to investigate multi-modal pretraining for learning complementary synergies between modalities that are beneficial for downstream semantic segmentation tasks.

RGB
Baseline difference map HHA Our difference map GT Fig. 9: Visualization of semantic segmentation results for the RGB-only baseline and our RGB-X approach, both of which are based on SegFormer-B4. "Acc" is short for pixel accuracy of the segmentation result. From left to right: RGB image, baseline difference map w.r.t. the ground truth, HHA image encoding depth information, our difference map, and segmentation ground truth.

APPENDIX A MORE IMPLEMENTATION DETAILS
We implement our experiments with PyTorch. We employ a poly learning rate schedule with a factor of 0.9 and an initial learning rate of 6e −5 . The number of warm-up epochs is 10. We now describe specific implementation details for different datasets. NYU Depth V2 dataset. We train our model with the MiT-B2 backbone on four 2080Ti GPUs, models with MiT-B4 and MiT-B5 backbones on three 3090 GPUs. The number of training epochs is set as 500. We take the whole image with the size 640×480 for training and inference. We use a batch size of 8 for the MiT-B2 backbone and 6 for MiT-B4 and MiT-B5. GPUs. The number of training epochs here is set as 32. We resize the input images to 480×480. We use a batch size of 12 for the MiT-B2 backbone and 8 for MiT-B4.
ScanNetV2 dataset. The model is trained on four 2080Ti GPUs. The number of training epochs here is set as 100. We resize the input RGB images to 640×480. We use a batch size of 12 for the MiT-B2 backbone.
Cityscapes dataset. The model is trained on eight A100 GPUs for 500 epochs. The batch size is set as 8. The images are randomly cropped into 1024×1024 for training and inference is performed on the full resolution with a sliding window of 512×512. The embedding dimension of the MiT-B4 backbone and MLP-decoder is set as 768. RGB-T MFNet dataset. The model is trained on four 2080Ti GPUs. We use the original image size of 640 × 480 for training and inference. Batch size is set to 8 for the MiT-B2 backbone and we train for 500 epochs. Consistent with the batch size of 8, the model based on MiT-B4 is trained on four A100 GPUs, which requires a larger memory. RGB-P ZJU dataset. The model is trained on four 2080Ti GPUs. We resize the image from 1224×1024 to 612×512. The number of training epochs is set as 400. We use a batch size of 8 for the MiT-B2 backbone and 4 for MiT-B4.
RGB-E EventScape dataset. The proposed model is trained with a batch size of 4 and the original resolution of 512×256 on a single 1080Ti GPU. The number of training epochs is set as 100.
The embedding dimension of the MiT-B4 backbone and MLPdecoder is set as 768.

APPENDIX B FAILURE CASE ANALYSES
In addition to the representative qualitative results and analyses that sufficiently show the effectiveness of CMX, we further present failure case analysis to help gather a more comprehensive understanding of RGB-X semantic segmentation. In Fig. 11, we show a set of failure cases in different sensing modality combination scenarios. The first rows shows that for the RGB-D semantic segmentation in a highly composite indoor scene with extremely densely arranged objects, the parsing results are still less visually satisfactory. In the second row of a nighttime scene, the guardrails are mis-classified by the RGB-X method as color cone, despite our model delivering more complete and consistent segmentation than the RGB-only model and having better segmentation of person with thermal properties. This illustrates that at night, the perception of some remote objects is still challenging in RGB-T semantic segmentation and it should be noted for safety-critical applications like automated driving. In the third row, the RGB-P model might be misguided by the polarized background area in an occluded situation, and yields less accurate parsing results, indicating that polarization, as a strong prior for segmentation of specular surfaces like glass and car regions, should be carefully leveraged in unconstrained scenes with a lot of occlusions. In the last row, the fences are partially detected as vehicles in the RGB-E segmentation result, but our model still yields more correctly identified pixels than the RGBonly model by harvesting complementary cues from event data.