{'fraction_non_alphanumeric': 0.033090185676392575, 'fraction_numerical': 0.009151193633952255, 'mean_word_length': 4.963226571767497, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multimodal Image Denoising based on Coupled Dictionary Learning

In this paper, we propose a new multimodal image denoising approach to attenuate white Gaussian additive noise in a given image modality under the aid of a guidance image modality. The proposed coupled image denoising approach consists of two stages: coupled sparse coding and reconstruction. The first stage performs joint sparse transform for multimodal images with respect to a group of learned coupled dictionaries, followed by a shrinkage operation on the sparse representations. Then, in the second stage, the shrunken representations, together with coupled dictionaries, contribute to the reconstruction of the denoised image via an inverse transform. The proposed denoising scheme demonstrates the capability to capture both the common and distinct features of different data modalities. This capability makes our approach more robust to inconsistencies between the guidance and the target images, thereby overcoming drawbacks such as the texture copying artifacts. Experiments on real multimodal images demonstrate that the proposed approach is able to better employ guidance information to bring notable benefits in the image denoising task with respect to the state-of-the-art.


INTRODUCTION
Image Denoising is a type of techniques that attenuate the noise in the corrupted images and at the same time preserve the details faithfully. Serving as a fundamental image processing operation, image denoising plays a critical role in various application scenarios such as object detection, image recognition and remote sensing [1]. Typical image denoising approaches that focus on single modality images have been thoroughly investigated, which can be categorized into two classes according to the image representation: spatial domain based local and nonlocal filters [2][3][4][5][6][7]; and (predefined or learned) transform domain based approaches [8][9][10][11][12].
However, in many practical application scenarios, it is commonly noticed that a certain scene is often imaged using various sensors that yield different image modalities. For example, in remote sensing domain, it is typical to have various image modalities of earth observations, such as a panchromatic band version, a multispectral bands version, and an infrared (IR) band version [13,14]. These different bands often exhibit similar textures, edges, corners, boundaries, or other salient features. In colorization [15] task, the output image has both chrominance channels and luminance channel which share consistent edges. These scenarios call for approaches that can capitalize on the availability of multiple image modalities of the same scene to address the image denoising task. A variety of joint image filtering approaches have been proposed to capitalize on the availability of additional guidance images, also referred to as side information [16,17], to aid the processing of target modalities [18][19][20][21][22][23]. The basic idea behind these methods is that the structural details of the guidance image can be transferred to the target image. However, these methods tend to introduce notable texturecopying artifacts, i.e. erroneous structure details that are not originally present in the target image because such methods typically fail to distinguish similarities and disparities between the different image modalities. This paper proposes a new effective multimodal image denoising approach based on coupled dictionary learning. With joint sparse representation induced by coupled dictionaries as the bridge, our approach incorporates a given clean guidance image as the side information to aid the denoising of the target image of interested modality. In particular, the proposed design has the ability to take into account both similarities and disparities between target and guidance images in order to deliver superior denoising performance. Further, we also incorporate self-similarity prior in our algorithm to enhance the denoising performance.

COUPLED IMAGE DENOISING
In this section, We introduce the proposed multimodal data model, the coupled image denoising and the coupled dictionary learning strategy.

Multi-modal Data Model
Given a vectorized noisy image X ns ∈ R N of one modality and a corresponding registered clean vectorized guidance image Y ∈ R N of different modality as side information. We first extract (overlapping) image patch pairs from this pair of multimodal images. In particular, let x ns i = RiX ns ∈ R n denote the i-th noisy image patch from X ns and let yi = RiY ∈ R n denote the corresponding i-th clean guidance image patch extracted from Y, where the matrix Ri is an n × N binary matrix that extracts the i-th patch from the image. Then, we propose a data model to capture the relationship -including similarities and disparities -between the two different modalities as follows: where sparse representation zi ∈ R k is common to both modalities, ui ∈ R k is specific to modality x, while vi ∈ R k is specific to modality y. In turn, Ψc and Φc ∈ R n×k are a pair of dictionaries associated with the common sparse representation zi, whereas Ψ and Φ ∈ R n×k are dictionaries associated with the specific sparse representations ui and vi, respectively. Note that the common sparse representation zi connects the patches of the two different modalities. The disparities between modalities x and y are distinguished by the sparse representations ui and vi, respectively. Parameter ∈ R n denotes additive zero-mean and homogeneous white Gaussian noise with the standard deviation σ.

Coupled Image Denoising
Assume that the coupled dictionaries Ψc, Φc, Ψ, Φ have been learned, then, based on the model (1) and (2), our coupled image denoising problem is addressed in two steps: coupled sparse coding and reconstruction of the denoised image.
Step 1: Coupled sparse coding. In this stage, we address a series of parallel sparse coding problems formulated as (3) where the standard deviation σ represents the noise level, and C is a constant. The objective with 0-penalty serves as the sparsityinducing regularizer. 1 The quadratic "data-fitting" constraint ensures that each pair of multimodal image patches are well approximated by their sparse representations with respect to the coupled dictionaries. In our case, we stick to 0 penalty for the sparse coding as it usually leads to better denoising performance than 1 penalty, which is also observed in [12,24].
Step 2: Reconstruction. Given the sparse codes zi, ui, the denoised image of interested modality can be estimated by solving (4), which leads to a closed form solution (5).
For µ = 0, this expression just represents the average of the denoised image patches on the overlapping areas, leading to a purified image. For non-zero µ, the estimation operation also refers to the original noisy image during the average.

Coupled Image Denoising with Group Sparsity
It is widely observed that natural images exhibit significant selfsimilarity property. The self-similarity prior is exploited to average 1 We denote by x 0 the number of nonzero elements of the vector x. Note, this 0 sparsity measure is a pseudo norm as it does not preserve the homogeneity property.
4: Unique dictionary training. Iterating between a) and b). 5: a) Update the sparse codes U as : where u j denotes the j-th row of U, ψ j denotes the j-th atom of dictionary Ψ, 6: b) Update the dictionary Ψ as : ; ∀j = 1, · · · , k. 7: Unique dictionary training for Φ is similar to Ψ but with u j , ψ j , X, Ψc replaced by v j , φ j , Y, Φc. 8: Return dictionaries Ψc, Φc, Ψ, Φ. out the noise among multiple similar patches [8,12]. Similar phenomena are also commonly observed in multimodal images, which motivates us to integrate self-similarity prior into our scheme via group sparsity regularization, which imposes that similar patch pairs share the same sparsity patterns in their representations.
The grouped-sparsity of a matrix A ∈ R 3k×l is defined as [u1, . . . , u l ], and [v1, . . . , v l ], respectively. In our case, we choose (p, q) = (0, ∞), leading to 0,∞ matrix pseudo norm, as a generalization of 0 vector pseudo norm. This amounts to restricting the number of nonzero rows of A.
To obtain groups of similar patches, we apply hierarchical clustering to separate all the patch pairs into M different clusters according to their similarity. Given the learned coupled dictionaries, sparse coding for the i-th cluster Si of patch pairs with a grouped-sparsity  (6) where |Si| denotes the cardinality of the cluster Si.

Coupled Dictionary Learning
The coupled image denoising require a group of coupled dictionaries to perform the sparse transform/decomposition task. These required coupled dictionaries are learned using our couple dictionary learning algorithm. Given a corpus of registered image patch pairs {(xi, yi)} T i=1 , our coupled dictionary learning problem is posed as follows: where, the first term with 2 norm promotes the fidelity of sparse representations to the signals and the regularization term with 1 norm promotes sparsity for the representations. The 2 norm constraints for each atom pair [ψcj; φcj] and individual atom ψj and φj are used to avoid trivial solution.
The coupled dictionary learning problems in (7) is a non-convex optimization problem. We solve it via an alternating optimization scheme that performs a) sparse coding and b) dictionary update alternatively. During the sparse coding stage, we fix the all the dictionaries and obtain the sparse representations, while during the dictionary updating stage, we fix the sparse codes and update the all the dictionaries. The dictionary updating formulations are adapted from Block Coordinate Descent algorithm [25], where we train the common dictionaries simultaneously while train the unique dictionaries individually, as shown in Algorithm 1.

EXPERIMENTS
We now present a series of experiments to validate the effectiveness of the proposed multimodal image denoising approach. The dataset 5 Fig. 3. Multimodal image denoising in terms of PSNR and RMSE with respect to noise level. We compare our basic approach (Ours) and the advanced version (Ours+) with state-of-the-art joint image filtering approaches, such as JBF [18], GF [19], SDF [20], JFSM [22] and DJF [21].
is from the EPFL infrared/RGB image database [26] 2 . Each pair of infrared image and RGB image in the dataset have been registered with each other. The target modality is near infrared images and the guidance modality is RGB images. As the response of near infrared band exhibit poor correlation with the response of the visible band, it is usually difficult to infer the brightness of an infrared image given a corresponding RGB version. Thus, it is challenging to take good advantage of the RGB modality to purify the noisy infrared version. The image pairs are randomly separated into two disjointed groups: training group (21 images) and testing group (8 images). We add zero-mean white Gaussian noise with different standard deviations σ = [4,8,12,16,20,24] into the testing infrared images to generate the noisy version. We compare our approach with state-of-the-art joint image filtering approaches, including Joint Bilateral Filtering (JBF) [18], Guided image Filtering (GF) [19], Static/Dynamic Filtering (SDF) [20], Deep Joint image Filtering (DJF) [21] and Joint Filtering via optimizing a Scale Map (JFSM) [22]. The same guidance images as in our approach are leveraged for these comparison approaches. We adopt the Peak Signal to Noise Ratio (PSNR) and the RMSE as the image quality evaluation metrics which are commonly used in the image denoising literature.

Coupled dictionary learning
We perform coupled dictionary learning on a clean dataset, i.e., a corpus of clean multimodal image patch pairs extracted from the clean training images. We adopt a common operation to construct a patch-based training dataset. First, the clean infrared images and the corresponding RGB images with only intensity information are divided into a set of √ n × √ n patch pairs. Then, we remove the mean from each patch, as the DC component is always preserved well during the denoising process. Finally, we vectorize these patches to form the training datasets {(xi, yi)} T i=1 of dimension n × T . Once the training dataset is prepared, we apply our coupled dictionary learning algorithm to learn the dictionary pairs [Ψc, Ψ] and [Φc, Φ] from the training datasets. The parameter setting is as follows: patch size n = 8, dictionary size k = 1024, training size T ≈ 50, 000, λ = 0.05. Figure 2 shows the learned coupled dictionaries from the corpus of clean infrared images and corresponding RGB version. We can find that any pair of atoms from common dictionaries Ψc and Ψc capture associated edges, blobs, textures with the same direction and location, as well as exhibit considerable resemblance and strong correlation to each other. This outcome indicates that the common dictionaries have indeed captured the similarities between infrared and RGB modalities. In contrast, the learned unique dictionaries Ψ and Φ represent the disparities of these modalities and therefore rarely exhibit resemblance.

Coupled image denoising
During the coupled image denoising phase, we evaluate basic approach with standard sparsity regularization and advanced version using group sparsity regularization to incorporate self-similarity prior. Figure 3 and 4 demonstrate the denoising performance, visual quality of the purified infrared images, as well as the corresponding error maps. As shown in these figures, our approach substantially attenuates the noise and, at the same time, reliably reserves image sharp details and suppresses artifacts. Therefore, the purified infrared images by our approach are cleaner and more visually plausible than the reconstruction by the state-of-the-art joint image filtering approaches. The visual quality is also demonstrated by the error maps where the denoised infrared images using our approach exhibit the least residual for different noise levels in comparison with the competing methods. In particular, it indicates that detailed structure information such as sharp edges, textures and stripes, can be effectively captured by learned coupled dictionaries. The average PSNR and average RMSE results for the multimodal image denoising task, shown in Table 1 and Figure 3, also confirm that our approach exhibits notable advantages over the competing methods. 3

CONCLUSION
This paper proposes a new effective multimodal image denoising approach based on coupled dictionary learning. The proposed approach explicitly incorporates sparsity prior, self-similarity prior and cross-similarity prior in the data model to captures the similarities and disparities between different image modalities in a learned sparse feature domain in lieu of the original image domain. This scheme is able to exploit a guidance image to aid the denoising of the target image of interested modality, achieving notable benefits in the task with respect to the state-of-the-art.