{'fraction_non_alphanumeric': 0.03337656996425069, 'fraction_numerical': 0.010408427979372963, 'mean_word_length': 4.964150943396226, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
\emph{cm}SalGAN: RGB-D Salient Object Detection with Cross-View Generative Adversarial Networks

Image salient object detection (SOD) is an active research topic in computer vision and multimedia area. Fusing complementary information of RGB and depth has been demonstrated to be effective for image salient object detection which is known as RGB-D salient object detection problem. The main challenge for RGB-D salient object detection is how to exploit the salient cues of both intra-modality (RGB, depth) and cross-modality simultaneously which is known as cross-modality detection problem. In this paper, we tackle this challenge by designing a novel cross-modality Saliency Generative Adversarial Network (\emph{cm}SalGAN). \emph{cm}SalGAN aims to learn an optimal view-invariant and consistent pixel-level representation for RGB and depth images via a novel adversarial learning framework, which thus incorporates both information of intra-view and correlation information of cross-view images simultaneously for RGB-D saliency detection problem. To further improve the detection results, the attention mechanism and edge detection module are also incorporated into \emph{cm}SalGAN. The entire \emph{cm}SalGAN can be trained in an end-to-end manner by using the standard deep neural network framework. Experimental results show that \emph{cm}SalGAN achieves the new state-of-the-art RGB-D saliency detection performance on several benchmark datasets.


I. INTRODUCTION
As an important research topic in computer vision and multimedia area, salient object detection (SOD) has attracted more and more attention in recent years. It aims at highlighting salient object regions from the given image and has been widely used in object-level applications in different fields, such as image understanding, object detection, and tracking. The main issues for the SOD task are twofold, i.e., 1) pixellevel representation and 2) saliency prediction/estimation. In the early years, many traditional methods have been proposed for saliency detection by exploiting some low-level feature representations, such as color, HOG, etc. In recent years, with the development of deep learning-based representation methods, salient object detection has been significantly improved via CNN based pixel-level representation. However, although the salient object detection has made great progress in recent years, it is still a challenging problem mainly due to the complicated background and different lighting conditions in the images.
Recently, many researchers attempt to utilize more modalities to boost the performance of SOD results. One popular way is to integrate the depth information into RGB saliency detection, which is known as RGB-D SOD problem. For this problem, both RGB and depth modalities are complementary to each other which can boost their respective representation and thus saliency prediction performance. As shown in Figure  1, the 'butterfly' object in RGB image is appeared better than that in the depth image, while the salient object in depth image shows better quality in the second row of Figure 1. Therefore, how to fuse these modalities (RGB, depth) adaptively is the key issue to the success of RGB-D salient object detection.
Most of previous works handle the multi-modal fusion problem by either serializing the RGB-D channels directly for data representation [1], [2], [3], [4] or processing the representation of each modality independently and then combining them together for the final multi-modal representation [5], [6], [7]. Although these strategies can obtain encouraging results, they are still difficult to fully explore cross-modal complementarity.
In this paper, we propose to address this issue by adopting a generative adversarial learning framework. We develop a novel cross-modality Saliency Generative Adversarial Network (cmSalGAN) for RGB-D salient object detection. Overall, cmSalGAN aims to learn an optimal modality-invariant and thus can fuse pixel-level representation of RGB and depth images together via a novel adversarial learning framework, which thus incorporates both information of intra-modality and correlation information of cross-modality images simultaneously for the RGB-D saliency detection problem. Specifically speaking, we use two encoder-decoder networks to extract the pixel-level features of RGB and depth images, respectively. We design a novel cross-modality adversarial learning mechanism to boost the representation learning of different modalities, which can achieve the purpose of information fusion. To further improve the detection results, the attention mechanism and edge detection module are also incorporated into cmSalGAN. The entire cmSalGAN can be trained in an end-to-end manner by using both saliency prediction cross-entropy loss and crossmodality adversarial learning loss.
Note that, generative adversarial networks (GANs) [8], [9] have been designed for image saliency detection tasks. Previous works [8], [9] focus on adversarial learning between saliency prediction and ground-truth salient object in the later stage. In contrast, cmSalGAN aims to conduct adversarial learning between different modality representations. That is, in cmSalGAN the generator and discriminator beat each other as a minimax game to learn discriminative common representation of heterogeneous multi-modality data for the final saliency prediction.
Overall, the main contributions of this paper are summarized as the following three aspects: • We propose to tackle the problem of cross-modality RGB-D image representation for saliency estimation by exploiting the generative adversarial representation learning. To the best of our knowledge, it is the first work to conduct multi-modality adversarial learning for RGB-D image representation. • A loss function is designed for cross-modality generative adversarial network (cmSalGAN) training to learn a discriminative common presentation for each pixel of RGB and depth images. • Comprehensive experiments on three widely used RGB-D benchmark datasets validate the effectiveness of the proposed cmSalGAN approach. It is also worthy to note that cmSalGAN achieves the new state-of-the-art performance on these benchmarks. The remainder of this paper is organized as follows. In section II, we briefly review some related works on RGB-D saliency detection and Generative Adversarial Networks (GANs). We present the detail of cmSalGAN in section III. In section IV, we implement cmSalGAN on several benchmarks to demonstrate the effectiveness of the proposed model.

II. RELATED WORK
In this work, we briefly review the related papers on RGB-D saliency detection and generative adversarial networks.

A. RGB-D saliency detection
Recently, many works have been proposed for the RGB-D saliency detection problem. The core aspect of these works is how to exploit the salient cues of both inter-modality and intramodality for the final saliency prediction. One kind of popular way is to first serialize the RGB-D channels directly for image representation and then conduct saliency prediction. For example, Chen et al. [1] propose a PCA network for RGB-D saliency detection which uses a novel complementarity-aware fusion module to deal with the complementarity of two modal information. In work [2], the authors propose a multi-modal fusion network with Multi-scale Multi-path and Cross-modal Interactions (MMCI) network for RGB-D saliency detection. The method aims to use a multi-scale multi-path manner to diversify the contributions of each modality by using a crossmodal interaction. Zhao et al. [3] propose a network named Contrast Prior and Fluid Pyramid integration (CPFP) for RGB-D saliency detection which integrates multi-scale cross-modal features by using a pyramid integration model. Chen et al. [4] recently propose Three-stream Attention-aware Network (TANet) for RGB-D saliency detection by using a novel tripletstream multi-modal fusion architecture to extract cross-modal complementary features.
Another way to handle the multi-modal fusion problem is first processing the representation of each modality independently and then combining them together for the final multimodal representation. For example, Han et al. [5] propose CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion (CTMF) network for RGB-D saliency detection. In CTMF [5], it first uses a two-stream architecture to exploit the multi-modal features for RGB and depth images respectively. Then, it aims to merges the representation of the two views to obtain the final saliency maps by using a multi-view CNN fusion model. In work [6], it first extracts handcrafted RGB and depth features in a two-stream network and then fuse them together for RGB-D saliency detection. Wang et al. [7] recently propose to employ the U-Net [10] framework to learn a switch map to estimate the weights for fusing RGB and depth saliency maps together. In addition, inspired by salGAN [8], Wang et al. [9] adopt Generative adversarial networks (GANs) for RGB-D saliency detection. They first use MSE and adversarial loss function to extract salient cues for RGB and depth modality, respectively. Then, they employ a reinforcement learning architecture to adaptively fuse these cues together for final saliency prediction.
Different from previous works [8], [9], our cmSalGAN aims to conduct adversarial learning between different modality representations and learns a kind of discriminative common representation for both RGB and depth data for saliency prediction.

B. Generative adversarial networks
Generative adversarial networks (GANs) [11] is originally proposed by Goodfellow et al. It has received increasing attention in the fields of machine learning and computer vision fields. Recently, GANs have been exploited for cross-modality visual data representation. For example, Dai et al. [12] propose a cmGAN network for cross-modality Re-ID task which uses GANs to learn feature representation from different modalities. Lekic et al. [13] employs GANs to fuse the radar sensor measurements with the camera images. Gammulle et al. [14] apply GANs for fine-grained human action segmentation. Li et al. [15] employ GANs to further enhance the retrieval accuracy. Similarly, Zhang et al. [16] propose SCH-GAN for semi-supervised cross-modal hashing representation. Dou et al. [17] design a cross-modal biomedical image segmentation network via an adversarial learning. Yu et al. [18] propose an edge-aware generative adversarial networks (EaGANs) for cross-modality MR image synthesis.   Recently, some works also employ GANs for saliency detection tasks. This is because the pixel-level measure of saliency results, such as binary cross-entropy loss, can be designed for per-pixel category prediction, as discussed in work [19]. However, this pixel-level model generally penalizes the false prediction on every pixel which thus lacks of explicitly modeling the correlation among adjacent pixels and may lead to local inconsistency and semantic inconsistency in the global saliency map prediction. Therefore, some researchers attempt to introduce some high-level evaluation criteria, such as adversarial network, to handle these issues. For example, works proposed in [8], [20], [21] and [9] all adopt adversarial learning mechanism and achieve better results on RGB or RGB-D related tasks. The adversarial learning mechanism judges whether a given saliency result is real or fake by the joint configuration of many label variables, and thus can enforce high-level consistency. Specifically, Fernando et al. [22] apply GANs for human saliency estimation to jointly model the contextual semantic and relations in different tasks. Pan et al. propose Saliency GAN (SalGAN) [8] for saliency prediction task which is trained with MSE and adversarial loss functions. SalGAN360 [23] further extends this framework for the 360 • image-based saliency prediction. Wang et al. [9] jointly use MSE and adversarial loss function for RGBD saliency detection.
Different from these adversarial learning based methods, we tackle the problem of cross-modality RGB-D image representation for saliency estimation by exploiting the generative adversarial representation learning. To our best knowledge, it is the first work to conduct multi-modality adversarial learning for RGB-D image representation and saliency detection problem, although multi-modality adversarial learning has been studied for RGB-I Re-ID task [12].

III. THE PROPOSED METHOD
In this section, we describe the detail of our cmSalGAN network. Figure 2 shows the overall of the proposed cmSalGAN network which mainly consists of the following four parts: • Two-stream Generator: In this paper, we adopt a twostream network as the generator to learn the feature representation for RGB and depth images respectively. This generator contains two encoder-decoder networks which share the initial weights and are trained to learn their respective network weight parameters. • Adversarial Feature Learning: Following the framework of generative adversarial learning, we introduce a discriminator, i.e. Adversarial Feature Learning (AFL) module, to make the features from the two branches combat to each other. This module will enhance the consistent constraint of our saliency detection framework from the perspective of feature learning. • Edge Detection Module: Inspired by existing works [24], [25] which adopt additional edge information for finegrained segmentation, we also utilize the edge features for more accurate saliency detection in our approach. • Saliency Prediction: We use the de-convolutional layer to restore the resolution of two modalities and feed them into the convolutional layer and ReLU layer to learn the parameters for adaptive fusion. Then, we employ a sigmoid activation function to generate the final saliency prediction. In the following subsections, we will present the details of each component as mentioned above.

A. Two-stream Generator
In the feature extraction phase, we utilize a two-stream generator that contains two encode-decoder networks to learn the deep representation of RGB and depth images, respectively. Specifically, the encoder module is a truncated ResNet-50 network (fully connected layers removed) with hole algorithm [26] which can keep the resolutions of feature maps unchanged. The weights of the encoder is initialized with a pretrained model on the ImageNet dataset [27] for object classification. The input samples are all resized into 224×224 and the output feature map Enc i from the 1st to 5th convolutional layer (Conv i , i ∈ {1, 2, 3, 4, 5}) has the resolution of 112, 56, 28, 28 and 28 respectively. The decoder contains deconvolutional layers which are used to increase the resolution of the encoded feature map. We use Decoder i , i ∈ {1, 2, 3, 4, 5} to denote the corresponding decoder layers. It is worthy to note that the input of depth image is preprocessed into three channels to make it consistent with the RGB branch before feed into the corresponding encoder network.
To obtain better feature representation, we follow the idea of U-Net [10] to fully utilize both low-level and high-level feature maps in the decoder network by using some skip network connections. This is because different layers contain different information, for example, the lower layers of encoder involve rich details while the high-level layers contains more semantic information. Therefore, we can obtain a kind of richer decoding feature map Dec i by fusing encoder's feature map Enc i and decoder's feature map Dec i−1 together. In addition, inspired by work [28], we employ both local and global attention schemes in the decoder network to obtain better feature representation. Specifically, in the global part, we use LSTM in Decoder 1 and Decoder 2 layers to obtain global context information by scanning the input feature maps along both horizontal and vertical directions. In global and local decoder layers, we use an attention mechanism to incorporate multi-scale context information. Given a convolutional feature map F ∈ R W ×H×C where W , H and C denote its width, height and number of channels respectively, we first use a convolutional layer (kernel size is 1 × 1) to transform it into a feature map with dimension K = W × H. Then, we extract the feature vector spatial location (w, h) and the dimension of this vector is K. Similar to previous work [28], we adopt Softmax function to normalize vector x w,h to obtain attention weights α w,h , the α w,h can be unfold to α w,h = (α w,h 1 , α w,h 2 , · · · , α w,h K ). Using the learned α w,h , we can obtain the attended feature F w,h attention in each spatial location (w, h) as where f i ∈ R C represents the feature vector at spatial location (w, h) in F . Thus we can get the final attention weighted feature F attention .

B. Adversarial Feature Learning
In this paper, we employ an adversarial learning to further explore cross-modal complementarity for RGB-D saliency detection task. Generally speaking, we introduce the idea of multi-view learning into the framework of generative adversarial networks [12]. More specifically, we take the output features of RGB and depth modality from the generator as an input of the convolutional (kernel size is 1 × 1) and sigmoid layer and generate saliency results from both modalities respectively. Then, we concatenate the predicted saliency results with RGB image and feed into the discriminator to judge the given results belong to RGB or depth modality. Through the adversarial multi-view learning, we can train the generator to learn a consistent feature representation of the RGB-D image.
The discriminator D used in our network is a standard convolutional network that contains convolutional layer, ReLU, max-pooling, fully connected layer and sigmoid activation layer. The detailed network architecture of discriminator used in this work is introduced in Table I. The discriminator is used to judge the given input belongs to the RGB modality or depth modality which will be beneficial for learning of discriminative feature representation. Concretely, after obtaining the feature maps Dec 5 rgb and Dec 5 depth from the two-stream network, we first employ a 1 × 1 convolutional operation on these feature maps and use a sigmoid activation function to obtain the final saliency map S r and S d whose resolution are all 112×112×1. To make the predicted saliency result consistent with the original image on the resolution, we conduct bilinear interpolation on the saliency map S r and S d and concatenate them with RGB image to form 4-channel feature maps, respectively. Finally, the feature maps are fed into the adversarial feature learning (AFL) module to achieve adversarial multiview learning across different modalities. During the training of our AFL module in which generator and discriminator compete with each other in the form of min-max game to learn the common representation of discriminant. Formally, the loss function of discriminator can be written as: where L denotes binary cross entropy loss, and D(·, ·) is the discriminator function used in the adversarial learning procedure. I r represents the corresponding original input RGB image. Here we use 1 to denote the target category of RGB sample and 0 for depth sample. Therefore, the final loss function of the proposed algorithm is formulated as where L BCE is defined as whereŜ ij , S ij represent the saliency map and corresponding ground truth, respectively.

C. Edge Detection Module
In order to estimate the final salient object more accurately, inspired by recent work [29], we further introduce an additional edge detection module on the basis of existing network to extract edge features and fuse them into our saliency prediction. Different from previous work [29], we integrate the residual convolutional blocks into the first three convolutional blocks of the encoder to realize feature transformation and edge feature encoding. Because the first three convolutional blocks of the encoder contain more detailed information and it will be more conducive to extract the edge information. Through the first three residual convolutional blocks, we can obtain three kinds of features and all of which have 16 channels. These features are concatenated together and fed into a 1 × 1 convolutional layer to generate the feature map whose dimension is 64. Note that the parameters of the generator are fixed when training the edge detection module and saliency prediction module.

D. Saliency Prediction.
After we obtain the convolutional features of two modalities, we upsample Dec 5 rgb and Dec 5 depth to make them have the same resolution via deconvolutional layers. Then, these two feature maps are concatenated together and fed into a convolutional layer and ReLU layer. Formally, the fused feature map can be transformed into saliency results via a 1 × 1 convolutional operation and sigmoid layer. In this paper, we adopt binary cross-entropy (BCE) loss to measure the distance between our saliency prediction and the ground truth saliency map. The loss function L BCE is defined as, (5) whereŜ ij , S ij represent the saliency map and corresponding ground truth, respectively.

IV. EXPERIMENTS
To evaluate the effectiveness of the proposed cmSalGAN approach, we test it on three benchmark datasets. In this section, we first introduce the datasets and evaluation metrics used in our experiments. Then, we present the implementation details of our cmSalGAN saliency detection algorithm. Finally, we compare our method with other state-of-the-art RGB-D saliency detection algorithms and further conduct the ablation studies for the proposed cmSalGAN model.

A. Datasets and Evaluation Metrics
Datasets. Three widely used RGB-D saliency detection benchmark datasets are used to evaluate our cmSalGAN method, i.e., NJUD [30], NLPR [31] and STEREO [32]. NJUD dataset [30] contains 2003 stereo images that are collected from the Internet, 3D movies and photographs acquired by a stereo camera. Then, the optical flow technique [33] is adopted to recover the depth maps. NLPR dataset [31] consists of 1000 images which are all taken by Kinect under different lighting conditions including both indoors and outdoors. The images in this dataset are selected from 5000 natural images and their depth maps and the saliency regions are annotated by five participants. STEREO dataset [32] consists of 797 stereo images.
For fair comparison, we adopt the same protocol to separate the dataset, as introduced in work [5]. The training subset contains 1400 and 650 samples from NJUD and NLPR dataset respectively. The validation set contains 100 samples from the NJUD and 50 samples from the NLPR dataset. The remaining samples in NJUD and NLPR and all the images in the STEREO dataset are used for testing purposes.
Evaluation Metrics: To achieve a comprehensive evaluation of the proposed cmSalGAN saliency detection method and other comparison detectors, we adopt four standard evaluation metrics to evaluate the predicted saliency maps, including Precision-Recall (PR) curve, S-measure scores, Mean Absolute Error (MAE) and maximum F-measure. Specifically, Precision-recall curve [34] is one of the most popular evaluation metric for saliency detection. The saliency result is first thresholded within the range of [0, 255] and the binarized saliency output is compared with the binary ground truth mask to obtain a pair of precision-recall values. S-measure [35] is a metric which evaluates both region-aware S region and objectaware S object structural similarity between saliency output and ground truth map, which can be formulated as:   [36] is defined as the average pixel-wise absolute difference between the saliency output and the ground truth map which is defined as The F-measure [37] is a balanced mean of average precision and average recall which is calculated as where β 2 is set as 0.3, as suggested in most previous works [9], [8]. The definition of precision and recall are: where TP, FP, TN and FN denote the numbers of true positives, false positives, true negatives and false negatives, respectively.

B. Implementation Details
Due to the limited images in the RGB-D saliency detection datasets, similar to previous related works [3], [2], [5], [4], [1], we adopt horizontal flip and random crop to augment the training data. In addition, we also add supervision with a smaller weight to each layer of the decoders so that being able to guide the decoder layers to learn better feature representation. The weight ratios of the five layers are set to {0.5, 0.5, 0.5, 0.8, 0.8} respectively. It is also worthy to note that the proposed Adversarial Feature Learning module is only used in the training phase to consistently learn the multi-modal adversarial features. That is, only the two-stream generator and feature fusion module are used in the test phase. Our model is implemented based on PyTorch and all experiments are implemented with a NVIDIA 1080Ti GPU. The Adam [38] optimizer is used for the training and the learning rate and batch size are set to 0.0001 and 4 respectively. Our cmSalGAN is an end-to-end network, which has no pre-training stage or other post-processing operations.
We compare our cmSalGAN with seven state-of-the-art deep learning based RGB-D salient object detection models including CTMF [5], PCA [1], MMCI [2], CPFP [3], TANet [4], GAN-RL [9] and AF [7]. In addition, we also report some traditional RGB-D salient object detection methods including DES [39], ACSD [30], DCMC [40], and LBE [41]. It is worth to note that part of our compared saliency detection algorithms do not release the code and we directly evaluate the saliency maps of corresponding algorithm provided by the authors, For comparison method CTMF [5], PCA [1], MMCI [2], TANet [4], ACSD [30], DES [39] and LBE [41], we directly evaluate the saliency maps of corresponding algorithm provided by the authors. For the other comparison methods, we run the original codes with default settings provided by the authors. Figure 3 shows the comparison results on PR curve. One can note that the proposed cmSalGAN achieves the best performance on all three datasets, especially on the NLPR dataset. This suggests the effectiveness of the proposed cmSalGAN method. Table II summarizes the comparison results on MAE, F-measure, and S-measure, respectively. Our proposed method generally performs better than other comparison methods on most of the evaluation measurements and datasets. This also fully validates the effectiveness and advantages of the proposed cmSalGAN method. More specifically, cmSalGAN outperforms GAN-RL [9] which also employs adversarial feature learning for RGB-D saliency detection. As discussed before, GAN-RL conducts adversarial learning for RGB and depth modality separately and adaptively fuses these results as post-processing. This obviously demonstrates the advantages of the proposed cross-view adversarial learning compared with adversarial feature learning on each modality separately in GAN-RL [9]. Figure 4 shows some qualitative results to better demonstrate the advantages of the proposed saliency detection method. Intuitively, cmSalGAN obtains the best saliency detection results compared with other approaches. Specifically, our approach can produce more fine-grained details as highlighted in the salient region as shown in the 1 st and 2 nd rows in Figure 4. We can also observe that the depth image in the 3 rd row contains some misleading salience cues that make it difficult to distinguish interferences in depth information. However, our method still works well in such challenging scenarios. These conclusions can also be drawn from the rest of the images.

D. Ablation Studies
In this section, we conduct some ablation studies to better understand the effect of each component in our model. Specifically speaking, we will first check the effect of dual-modality fusion. Then, we will discuss the effect of adversarial feature learning module and edge information. Finally, we will discuss the limitations of the proposed RGBD salient object detection algorithm.
Effectiveness of dual-modality fusion. We compare the proposed approach with only one modality used version to validate the effectiveness of the fusion module for RGB-D saliency detection. As shown in Table III, it is easy to find that the result of fused RGB-D saliency detection is significantly better than only one modality used version on all the three datasets. For example, the results of G depth and G RGB achieve 0.8239 and 0.8560 on F-measure on the NJUD dataset respectively, while the G RGB+depth method which fuse RGB and depth features together achieves 0.8854 which is a significant improvement. Similar conclusions can also be drawn from other evaluation metrics and benchmark datasets. These experiments all validated the effectiveness of dual-modality fusion for saliency detection.
Effectiveness of AFL and Edge components. To validate the effectiveness of AFL and Edge components in our network, we implement three variations of the model, i.e.,  1) cmSalGAN-Edge-AFL that remove Edge and AFL module, only use BCE loss function in the training phase.
3) cmSalGAN that add Edge module. As shown in Figure 5, we can note that, (1) Only use BCE loss function, the saliency detection results dropped in all three benchmark datasets. (2) Add AFL module could improve the performance of our saliency detection network, which validate the effectiveness of the AFL module. We can obtain 0.8932, 0.9037 and 0.8912 on the F-measure on the NJUD, NLPR, and STEREO as shown in Table II. And it achieves the state of the art performance. (3) We introduce an edge detection module in the shallow layer of the encoder could further improve the performance of our cmSalGAN network, which validate the effectiveness of the Edge module. When integrating the edge feature into our model, we can obtain 0.8965, 0.9070, 0.8938 on the F-measure on the NJUD, NLPR, and STEREO. And achieve better saliency detection performance.
We also conduct a visual comparison of saliency maps generated by these three variants of our model in Figure  6. From both qualitative and quantitative analysis, we can observe that the proposed adversarial feature learning module can significantly improve the deep representation learning for RGB-D saliency detection. And integrate the edge feature into our model could further improve the performance of the saliency detection network.

V. CONCLUSION AND FUTURE WORKS
In this paper, we design an end-to-end RGB-D salient object detection framework based on generative adversarial feature learning. Our framework utilizes a two-stream generator to learn the feature representations of RGB and depth, respectively. We creatively fuse the features of RGB and depth with a feature embedding module to handle the limitations of single modality. More importantly, we conduct the adversarial feature learning between both RGB and depth modalities to boost their deep representations in the training stage. Our experiments validated the effectiveness of the proposed method on multiple saliency detection benchmarks.
Note that, the proposed cmSalGAN provides a general framework for RGB-D saliency detection tasks. One can incorporate any backbone network into our cmSalGAN. In our future works, we will consider utilizing lightweight neural networks, such as ShuffleNet [42] and MobileNet [43], to improve the efficiency of the proposed RGB-D saliency detection algorithm.