{'fraction_non_alphanumeric': 0.04015215553677092, 'fraction_numerical': 0.01828940290478752, 'mean_word_length': 4.9722349701698025, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network

Salient object detection is the pixel-level dense prediction task which can highlight the prominent object in the scene. Recently U-Net framework is widely used, and continuous convolution and pooling operations generate multi-level features which are complementary with each other. In view of the more contribution of high-level features for the performance, we propose a triplet transformer embedding module to enhance them by learning long-range dependencies across layers. It is the first to use three transformer encoders with shared weights to enhance multi-level features. By further designing scale adjustment module to process the input, devising three-stream decoder to process the output and attaching depth features to color features for the multi-modal fusion, the proposed triplet transformer embedding network (TriTransNet) achieves the state-of-the-art performance in RGB-D salient object detection, and pushes the performance to a new level. Experimental results demonstrate the effectiveness of the proposed modules and the competition of TriTransNet.


INTRODUCTION
Salient object detection (SOD) simulates the visual attention mechanism to capture the prominent object in the scene. It has been widely applied in the computer vision tasks, such as image segmentation [18], tracking [30,47,83], retrieval [25], compression [32], edit [65] and quality assessment [34].
As a pixel-level dense prediction task, salient object detection usually uses CNN based U-Net framework [58] (Fig. 1(a)) to encode images from low-level to high-level, and then decode back to the full spatial resolution. Research [74] points out that the performance tends to saturate quickly when gradually aggregating features from high-level to low-level. In other words, high-level features contribute more to the performance. Therefore, we propose a triplet transformer embedding module (TTEM) to enhance the feature representation of high three layers.
As we all known, Transformer [62] has recently attracted a lot of attention in computer vision domain, but it is also encountering high computational cost problem. PVT [66] adopts a spatialreduction attention (SRA) layer to reduce the resource cost to learn high-resolution feature maps. CvT [72] introduces convolutional into the Vision Transformer architecture to concurrently maintain a high degree of computational and memory efficiency. Swin Transformer [44] uses the shifted windows calculation method to propose a hierarchical Transformer, which has the flexibility of modelling at various scales and has linear computational complexity relative to the image size. Multi-Scale Vision Longformer [82] proposes multi-scale coding structure, and further improves its attention mechanism to reduce the computational and memory cost.
Unlike these profound designs, we introduce Transformer into U-Net framework to enhance the features of high three layers, which can be easily integrated into existing U-Net framework for significant improvement with less cost. The features of high three layers show the different attributions but the same in nature, which are the different aspects of the same input image. The proposed triplet transformer embedding module (TTEM) is composed of three standard transformer encoders [62] with shared weights. It is beneficial to find the common information which is hidden in the multi-level features and achieve the better fusion by learning long-range dependencies across levels.  Taking TTME as the core, we further propose the triplet transformer embedding network (TriTransNet). At first, multi-level features are adjusted to the same size by a transition layer and progressively upsampling fusion module. Second, features are fed into TTME to be enhanced. Last, the output features of TTME and the features of low two layers are effectively fused by a three-stream decoder.
Our proposed TriTransNet is the first attempt to use three standard transformer encoders with shared weights to enhance the feature representation. Different from visual-transformer-FPN (VT-FPN) [71] in Fig. 1(b) which merges visual tokens from feature map of each layer with one transformer, our TriTransNet in Fig. 1(c) adopts weight sharing strategy to make visual tokens extracted from multi-level features more abundant enough to express the original information, and meanwhile high-layer semantic information and middle-layer texture or shape information are both better excavated by parallel self-attention mechanism.
In addition, depth information is proved to supply the useful cues and boost the performance for saliency detection [51], especially in some challenging and complex scenarios, e.g. the low color contrasts between salient objects and background, the cluttered background interferences. But depth image with poor quality, which likes a noise, brings some negative influences [22]. Following depth guided manners [11,12,23,52,57,60,79,89,93] we design depth purification module, which uses depth information to purify the color features.
Our main contributions can be summarized as follows: • A triplet transformer embedding module is proposed and embedded into CNN based U-Net framework to enhance the feature representation. It is composed of three standard transformer encoders with shared weights, learning the common information from multi-level features. • Based on the proposed triplet transformer embedding module, triplet transformer embedding network is designed to detect the salient objects in RGB-D image. Multi-level features from encoder need to be adjusted to the same size by a transition layer and progressively upsampling fusion module, and then fed into triplet transformer embedding module. Then the output of triplet transformer embedding module need to be combined with the features of low two layers by three-stream decoder to achieve the decoding process. • Depth image is viewed as the supplement to color feature, and attached to color feature to enhance the feature representation by depth purification module which introduces spatial attention and channel attention. • Due to the advantage of the proposed triplet transformer embedding module, the proposed model pushes the performance of RGB-D salient object detection to a new level and shows the state-of-the-art performance on several public datasets.

RELATED WORK 2.1 RGB-D saliency detection
In RGB-D image, color image provides appearance and texture information, and depth image contains 3D layout and spatial structure. The fusion of color feature and depth feature is always an important issue in RGB-D saliency detection. References [10,45,85,90] use early fusion or input fusion, references [7,11,38,39,41] employ two-stream subnetwork to achieve the middle fusion, references [53,57,73,89,93] apply depth guided fusion and references [14,46,64] adopt late fusion. Although depth information can supply the useful cues for saliency detection [51], depth image with poor quality can bring some negative influences too [22]. In order to solve the filtering issue of low-quality depth map, D3Net [22] uses gate mechanism to filter the poor depth map, EF-Net [9] enhances the depth maps by color hint map, DQSD [5] integrates a depth quality aware subnetwork into the classic bi-stream structure, assigning the weight of depth feature before conducting the fusion. In addition, CoNet [33], DASNet [88], SSDP [68] and MobileSal [73] introduce depth estimation, learning to detect the salient object simultaneously.
In the paper, we adopt depth guided manner. Depth information is viewed as the supplement to the color feature. It enhances the color feature by attention mechanism.

Transformer
Transformer is first proposed by [62] to replace recurrent neural networks (RNN), e.g.long short-term memory (LSTM) and gated recurrent unit(GRU) for machine translation tasks. It can overcome intrinsic shortages of RNN and has dominated nature language processing (NLP) field and are becoming increasingly popular in computer vision tasks, e.g. image classification [19], object detection [4], semantic segmentation[? ], line segment [77], person re-identification [94], action detection [87], image completion [91], 3D point cloud processing [26,86], pose estimation [59], facial expression recognition [48], object tracking [49] etc. DETR [4] takes the lead in applying Transformer to the field of object detection and achieves the better performance. The successful use of ViT [19] in image classification tasks has made the research on visual Transformer a hot topics. SETR [92] deploys a pure Transformer as the encoder, combined with a simple decoder, to achieve a powerful semantic segmentation model. Besides, TransUNet [8] uses the pretrained ViT [19] as a powerful backbone of the U-Net [58] network structure, and performs well in the field of medical image segmentation.
However, pure transformer has great limitations. As a result, many improved visual transformers have emerged. The Conditional Position encodings Visual Transformer (CPVT) [17] replaces the fixed position encoding in ViT [19] with the proposed conditional position encoding (CPE), which makes it possible for Transformer to process inputs of arbitrary sizes. Tokens-to-Token (T2T) [78] adopts a novel progressive tetanization mechanism, which models local structural information by aggregating adjacent tokens into one token, while reducing the length of the token. LocalViT [42] adds locality to vision transformers by introducing depth-wise convolution into the feed-forward network, improving a locality mechanism for information exchange within a local region. Considering that most visual Transformers ignore the inherent structural information inside the sequence of patches, Transformer-iN-Transformer (TNT) [27] proposes to use outer Transformer block and inner Transformer block to model patch-level and pixel-level representations, respectively. Co-Scale Conv-Attentional Image Transformers [76] designs a conv-attention module to realize relative position embedding and enhance computation efficiency, and further proposes a co-scale mechanism to introduce cross-scale attention to enrich multi-scale feature.
On the other hand, CNN has the advantages of extracting low level features and strengthening locality, while Transformer has the advantages in establishing long-range dependencies. Some research makes full use of both advantages. TransFuse [84] uses a dual-branch structure, which uses Transformer to capture global dependencies, while low-level spatial details are extracted by CNN branches. Similarly, CoTr [75] uses the CNN backbone to extract feature representations and proposes to use deformable Transformer (DeTrans) to model long-range dependencies, effectively bridging the convolutional neural network and Transformer. ICT [63] uses transformer to recover pluralistic coherent structures together with some coarse textures, and uses CNN to enhances local texture details of coarse priors, so as to achieve excellent results on the image completion task. TransT [13] uses Siamese-based CNN network for feature extraction, and designs the self-attention-based ego-context augment (ECA) and cross-attention-based cross-feature augment (CFA) modules for feature fusion. Compact Transformers [28] eliminates the requirement for class token and position embedding through a novel sequence pooling strategy and the use of convolutions, so as to perform head-to-head with state-of-the-art CNNs on small datasets.
Follow this strategy, we present triplet transformer embedding module which is embedded into a U-Net framework to improve the performance of RGB-D saliency detection. Combining both advantages, our model achieves the state-of-the-art performance.

PROPOSED METHOD 3.1 Overview
The overall framework of the proposed triplet transformer embedding network is depicted in Fig.2(a), which consists of multi-modal fusion encoder, feature enhancement module and three-stream decoder. The details can be seen in the following sections.

Multi-modal fusion encoder
Color and depth image in RGB-D image are two expressions for different modalities of the same scene. Color image provides appearance cue and depth image shows three dimension spatial information. Due to existence of poor quality depth map induced by the imaging devices or conditions, we propose multi-modal fusion encoder, in which depth features are first purified by multi-modal features using attention mechanism, and then served as supplement to the color feature by the residual connection [29]. Residual part is designed as depth purification module (DPM), and shortcut connection part is used to preserve more original color information.
In DPM which is shown in Fig. 2(b), depth feature is concatenated with color feature, and fed into a channel attention module to get attentive channel mask, which is used to purify the depth feature in a channel manner. Next, purified depth feature is fed into a spatial attention module again to generate attentive spatial mask, which is used to purify the depth feature in a spatial manner. The process can be described as: where and represent color and depth features extracted by backbone network respectively in which = 1, · · · , 5, (·) denotes concatenation and following convolution operation, (·) and (·) are channel and spatial attention operation which is proposed by CBAM [70], "×" is element-wise multiplication operation, "+" is element-wise addition operation.
Thus, the depth feature with poor quality can be purified, and then attached to color feature to generate more accuracy feature representation ( = 1, · · · , 5).

Feature enhancement module
In this module, we first adjust the features of high three layers to the same size, and then use the triplet transformer embedding module to enhance the feature representation by learning longrange dependency across levels, and last concatenate the input and output of triplet transformer embedding module to preserve more original information.
3.3.1 Scale adjustment module. The triplet transformer embedding module is composed of three standard transformer encoders with shared weights. Its input should be the features with the same size. But the sizes of the multi-level features from multi-modal fusion encoder are the different. Therefore, the first important task is to adjust the sizes of multi-level features.
At first, a transition layer which contains a 3×3 convolution and a ReLU activation function is applied on . It can adjust the number of channels of multi-level features to the same size. It can be described as: where (·) is 3×3 convolution operation, and (·) is ReLU activation function.
Then, we design a progressively upsampling fusion module which is used to adjust the resolution of the features in the high three layers to the same size. Since the direct upsampling with 2× or 4× ratio will bring some noises, the features are progressively upsampled and fused. The fusion process can be described as: (·) is shown in Fig.2(d). The detail can be described as: where ℎ ℎ and denote the feature from the higher layer with low resolution and the feature from the lower layer with high resolution, respectively, and (·) denotes 2×upsampling operation.
Compared with direct 2×, 4×upsampling on ′ 4 and ′ 5 , progressively upsampling fusion module can not only adjust the features to the same resolution but also increase the spatial detail of feature in the high layer by progressive fusion process.
Thus, the features ( = 3, · · · , 5) with the same scales will be served as the input and fed into next triplet transformer embedding module.

Triplet
Transformer Embedding Module (TTEM). The features are first converted into the sequences of feature embedding, and then fed to three standard transformer encoders with shared weights to model the long-range relationship among different levels, and last reshaped to the original size of features.
Specifically, each input feature ( = 3, · · · , 5) are first flattened into a 1D sequence { | = 1, · · · , }, where is the number of patches. Each patch is then mapped into a latent -dimensional embedding space by a trainable linear projection layer. Furthermore, we learn specific position embedding which are added to the patch embedding to retain positional information. The process can be described as: where = { | = 1, · · · , } is a 1D learnable positional embedding.
The remaining architecture essentially follows the standard transformer encoder [62] which stacks transformer layer. It is shown in Fig.2(c). Each transformer layer contains multi-headed self-attention (MSA) and multi-layer perceptron (MLP) sublayer. Layer normalization (LN) [2] are inserted before these two sublayers, and the residual connection is performed after these two sublayers. The process can be described as: where denotes the number of transformer layers in the standard transformer encoder.

Feature concatenation module.
The outputs of three weights shared transformer encoders ( = 3, · · · , 5) fuses the information of three layers by Transformer mechanism, so as to enhance the original feature representation. In order to preserve the more original information, we further cascade these outputs with original features to generate the enhanced features of high three layers. The process can be described as:

Three-stream decoder
After the features of high three layers are enhanced by the proposed triplet transformer embedding module, we will combine them with the features of low two layers to achieve the decoding process. There are two decoding methods. One is single-stream decoding and the other is three-stream decoding. The single-stream decoding first fuses three output results of feature enhancement module, and then combine it with two features in the low layers. The three-stream decoding first combines each output result of feature enhancement module with two features in the low layers, and then fuses threestream results. We conduct two decoding processes, and find threestream decoding is better than single-stream decoding. Next, we use formula to show three-stream decoder as follow: The above three features are performed upsampling, convolution operation and sigmoid function to generate the saliency maps ( = 1, · · · , 3) which are supervised by the ground truth maps.
At last, we also fuse all the features above to generate the final saliency map.
Pixel position aware loss [69] is adopted for end-to-end training. The whole loss is defined as: where is ground truth saliency map.

EXPERIMENTS 4.1 Datasets and evaluation metrics
4.1.1 Datasets. We evaluate the proposed method on six challenging RGB-D SOD datasets. NLPR [54] includes 1000 images with single or multiple salient objects. NJU2K [36] consists of 2003 stereo image pairs and ground-truth maps with different objects, complex and challenging scenes. STERE [50] incorporates 1000 pairs of binocular images downloaded from the Internet. DES [15] has 135 indoor images collected by Microsoft Kinect. SIP [22] contains 1000 high-resolution images of multiple salient persons. DUT [56] contains 1200 images captured by Lytro camera in real life scenes. For the sake of fair comparison, we use the same training dataset as in [11,22], which consists of 1,485 images from the NJU2K dataset and 700 images from the NLPR dataset. The remaining images in the NJU2K and NLPR datasets and the whole datasets of STERE, DES and SIP are used for testing. In addition, on the DUT dataset, we follow the same protocols as in [33,39,56,57,90] to add additional 800 pairs from DUT for training and test on the remaining 400 pairs. In summary, our training set contains 2,185 paired RGB and depth images, but when testing is conducted on DUT, our training set contains 2,985 paired ones.

Evaluation Metrics.
We adopt five widely used metrics to evaluate the performance of our model and other state-of-the-art RGB-D SOD models, including the precision-recal(PR) curve [3], E-measure [21], S-measure [20], F-measure [1] and mean absolute error (MAE) [55]. Specifically, the PR curve plots precision and recall values by setting a series of thresholds on the saliency maps to get the binary masks and further comparing them with the ground truth maps. The E-measure simultaneously captures global statistics and local pixel matching information. The S-measure can evaluate both region-aware and object-aware structural similarity between saliency map and ground truth. The F-measure is the weighted harmonic mean of precision and recall, which can evaluate the overall performance. The MAE measures the average of the perpixel absolute difference between the saliency maps and the ground truth maps. In our experiment, E-measure and F-measure adopts adaptive values.

Implementation details
During the training and testing phase, the input RGB and depth images are resized to 256×256. Multiple enhancement strategies are used for all training images, i.e. random flipping, rotating and border clipping. Parameters of the backbone network are initialized with the pretrained parameters of ResNet-50 network [29]. The hyper-parameters in transformer encoder are set as: = 12, = 768, = 1024. The rest of parameters are initialized to PyTorch default settings. We employ the Adam optimizer [37] to train our network with a batch size of 3 and an initial learning rate 1e-5, and the learning rate will be divided by 10 every 60 epochs. Our model is trained on a machine with a single NVIDIA GTX 3090 GPU. The model converges within 150 epochs, which takes nearly 15 hours.

Comparisons with the state-of-the-art
Our model is compared with 16 state-of-the-art RGB-D SOD models, including D3Net [22], ICNet [41], DCMF [6], DRLF [67], SSF [81], SSMA [43], A2dele [57], UCNet [80], CoNet [33], DANet [90], JLDCF [24], EBFSP [31],CDNet [35], HAINet [40], RD3D [10] and DSA2F [61]. To ensure the fairness of the comparison results, the saliency maps of the evaluation are provided by the authors or generated by running source codes. Figure.3 shows the comparison results on PR curve. Table.1 shows the quantitative comparison results of four evaluation metrics. As can be clearly observed from figure that our curves are very short, which means that our recall is very high. Furthermore, from the table, we can see that all the evaluation metrics are nearly the best on six datasets, so as to verify the effectiveness and advantages of our proposed method. Only two S-measure values in NLPR and STERE datasets are inferior to the best, but they are also the second best. Combined with the results of figure and table, our method achieves the impressive performance.

Qualitative Evaluation.
To make the qualitative comparisons, we show some visual examples in Figure.4. It can be observed that our method has better detection results than other methods in some challenging cases: similar foreground and background(1 -2 rows), complex scene(3 -4 ℎ rows), low quality depth map(5 ℎ -6 ℎ rows), small object(7 ℎ -8 ℎ rows) and multiple objects(9 ℎ -10 ℎ rows). These indicate that our approach can better locate salient objects and produce more accurate saliency maps. In addition, our approach can produce more fine-grained details as highlighted in the salient region(11 ℎ -12 ℎ rows). This is also the proof of the effectiveness of our method.

Ablation studies
We conduct ablation studies on NLPR, NJU2K, SIP and STERE datasets to investigate the contributions of different modules in the proposed method.  Table. 2. Further, we replace TTEM with gated recurrent unit (GRU) [16], whose result is shown in the variant No.2 of Table. 2. The variant No.3 of Table. 2 is the result of siamese transformer applied in the high two layers. The variant No.4 of Table. 2 is the result of quadruplet transformer applied in the high four layers. The variant No.5 of Table. 2 is our result of triplet transformer applied in the high three layers.
It can be clearly observed that compared with No.1, the result of our TriTransNet is improved 0.016 in the S-measure metric, 0.021 in the F-measure metric, 0.008 in the E-measure metric and 0.007 in the MAE metric on average. Meanwhile, compared with No.2, the result of our TriTransNet is improved 0.012 in the S-measure metric, 0.014 in the F-measure metric, 0.006 in the E-measure metric and 0.005 in the MAE metric on average. TTEM plays an important role in the performance improvement.
In addition, we compare No.3, No.4 and No.5 and find that Triplet win Siamese in S-measure, F-measure, E-measure, and MAE about 0.009,0.016,0.006 and 0.005 on average, and outperform Quadruplet about 0.010,0.009,0.005 and 0.004 on average. Our TriTransNet enhances long-range dependency of semantic information by using the features in the high three layers, and further combines with three-stream usampling decoding in the low two layers to perfectly depict the detailed boundary, so as to achieve the best performance.

4.4.2
The effectiveness of three-stream decoder. we further conduct the ablation study by replacing three stream decoder with singlestream decoder to check the effectiveness of the designed threestream decoder.

CONCLUSIONS
In the paper, we introduce transformer into U-Net framework to detect salient object in RGB-D image. Different from existing combination method of transformer and convolutional neural networks, we propose a triplet transformer embedding module which can be embedded into existing U-Net models for the better feature representation by learning long-range dependency among different levels with less cost. Furthermore, we use depth information to enhance RGB features by depth purification module. Experimental results show our method pushes the performance to a new level,  and ablation studies also verify the effectiveness of each module. In the future, we will achieve the same task by a pure transformer, and further discuss their respective advantages to achieve the better combination.