{'fraction_non_alphanumeric': 0.034827917180858356, 'fraction_numerical': 0.010580922345628229, 'mean_word_length': 4.790525340743681, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LADMM-Net: An Unrolled Deep Network For Spectral Image Fusion From Compressive Data

Image fusion aims at estimating a high-resolution spectral image from a low-spatial-resolution hyperspectral image and a low-spectral-resolution multispectral image. In this regard, compressive spectral imaging (CSI) has emerged as an acquisition framework that captures the relevant information of spectral images using a reduced number of measurements. Recently, various image fusion methods from CSI measurements have been proposed. However, these methods exhibit high running times and face the challenging task of choosing sparsity-inducing bases. In this paper, a deep network under the algorithm unrolling approach is proposed for fusing spectral images from compressive measurements. This architecture, dubbed LADMM-Net, casts each iteration of a linearized version of the alternating direction method of multipliers into a processing layer whose concatenation deploys a deep network. The linearized approach enables obtaining fusion estimates without resorting to costly matrix inversions. Furthermore, this approach exploits the benefits of learnable transforms to estimate the image details included in both the auxiliary variable and the Lagrange multiplier. Finally, the performance of the proposed technique is evaluated on two spectral image databases and one dataset captured at the laboratory. Extensive simulations show that the proposed method outperforms the state-of-the-art approaches that fuse spectral images from compressive measurements.


Introduction
A hyperspectral (HS) image can be considered as a three-dimensional (3-D) data set that contains the light reflective responses of a two-dimensional (2-D) scene across tens or hundreds of spectral bands ranging from the visible spectrum (VIS, 400−700 nm) to the shortwave infrared region (SWIR, 700−2400 nm) [1,2].
HS images provide detailed spectral information of the scene, enabling the identification of distinct materials included in the covered region. These images have been considered for various applications such as precision agriculture, environment monitoring, and medical diagnosis [3,4]. HS imaging sensors commonly capture low-spatial-resolution data sets with the goal of achieving a high signal-to-noise ratio (SNR) in the acquired observations. To overcome this drawback, data fusion has emerged as a signal processing task that focuses on merging the information in HS images with the information provided by the high-spatial-resolution but low-spectral-resolution multispectral (MS) images [5,6,7].
Fusion methods are typically applied to HS and MS images captured by scanning sensors whose functioning is based on the Nyquist-Shannon sampling theorem [8,9]. Nevertheless, scanning sensors require a huge amount of observations to capture the relevant information in the covered scenes. In this regard, compressive spectral imaging (CSI) has emerged as an alternative sensing framework that captures the relevant information embedded in spectral images using a reduced set of projections [10]. The coded aperture snapshot spectral imaging (CASSI) is the most representative CSI system whose measurements basically are projections of encoded versions of the input spectral field [11]. Multiple variants of the CASSI system have been developed including the three-dimensional CASSI (3D-CASSI) [10], the colored CASSI (C-CASSI) [12] and the snapshot colored CSI (SCCSI) [13], among others. Various image fusion methods from multisensor compressive measurements have been recently proposed. More precisely, these methods determine the acquisition model that describes the multi-sensor optical system and formulate an optimization problem.
Then, an iterative algorithm is developed to estimate the high-resolution spectral image from compressive samples [14,15,16]. These methods assume that high-resolution spectral image can be sparsely represented on a predefined transform basis and they also exhibit high running times. On the other hand, various deep learning approaches have been recently developed providing state-of-the-art performances for different imaging applications including hyperspectral image analysis [17] and target detection in SAR imagery [18,19].
However, deep learning approaches typically suffer from a lack of explainability of the inverse mapping that recovers the image of interest from degraded measurements [20,21].
In this regard, deep algorithm unrolling is an approach for solving inverse problems whose performance and interpretability have attracted the attention of the signal and image processing community in the past decade. This approach takes advantage of both the interpretability of the model-based iterative algorithms and the remarkable performance of deep neural networks [21]. Specifically, this approach casts each update step of an iterative algorithm into a network-based structure whose concatenation describes a deep network.
The first work in deep algorithm unrolling was proposed by Gregor and LeCun in the context of sparse coding [22]. Basically, this method maps each update of the iterative soft-thresholding algorithm (ISTA) to a network-based structure that consists of learning the inversion induced by two operators to compute the output estimate. This technique is referred to as Learned ISTA (LISTA). A second approach based on the ISTA algorithm is the ISTA-Net architecture [23] that has been designed to recovering images captured under the compressive sensing (CS) scheme. This approach exploits the representation power of the convolutional neural networks (CNN) to build a sparsity-inducing nonlinear transform for representing the target image. Furthermore, authors introduce an enhanced version, called ISTA-Net+, that relies on the fact that the image residuals can be better described in a sparsity-promoting dictionary. Recently, the ADMM-CSNet has been reported in [24,25] for reconstructing images from CS samples. In essence, this work proposes two deep learning architectures that unroll two versions of the ADMM algorithm, where the matrix inversions are computed using techniques based on the fast Fourier transform (FFT). Other unrolling approaches for solving inverse problems has been also introduced such as the model-based deep learning (MoDL) architecture [26], the FBPConvNet [20], and the analytic compressive iterative deep (ACID) framework [27]. This paper focuses on developing a network architecture based on the linearized version of the ADMM algorithm [28] to solve the image fusion problem from HS and MS compressive measurements. This algorithm unfolding-based network, called LADMM-Net, is an interpretable architecture that exploits the sparsityinducing nonlinear transform to estimate the image high-frequency content embedded in both auxiliary and dual variables of the ADMM algorithm [29]. In essence, the proposed architecture casts each iteration of the LADMM algorithm into a network-based processing layer whose cascading deploys a deep network. Every layer consists of an approximation unit (AU) followed by a network-based refinement unit (NRU). The AU considers the information embedded in the coded aperture patterns as well as the image high-frequency content computed by the previous layer to obtain an estimate of the fused image. In contrast, the NRU is a network-based structure that can be seen as a nonlinear transform function relied on a CNN that estimate the high-frequency information of the target spectral image. The main contributions of this work are synthesized as follows.
1. The proposed approach dubbed LADMM-Net basically is a deep learning architecture that solves the data fusion problem from HS and MS compressive measurements. In particular, this approach casts each iteration of a linearized version of the ADMM algorithm into a CNN structure whose cascading describes a deep network. Moreover, the linearized version of the ADMM allows to estimate approximations of the variable of interest without resorting to costly matrix inversions. In addition, the information embedded in both the auxiliary variable and the Lagrange multiplier is obtained by the network-based structure to remarkably improve the fusion estimation.
2. Secondly, the performance of the proposed architecture is evaluated for two spectral image databases and one real data set captured in the laboratory. Furthermore, the proposed deep network is tested for different processing layer numbers and distinct compression ratios.
3. Extensive simulations show that that the proposed deep learning architecture outperforms other stateof-the-art methods that obtain high-resolution images from HS and MS compressive measurements.
Since data fusion can be considered a particular inverse problem, the proposed architecture is adapted for reconstructing images from compressive random projections.

Related work
Recently, various algorithm unrolling methods have been developed for solving inverse problems [21]. In this regard, LISTA is considered the first deep algorithm unrolling technique that maps each ISTA iteration into a fully connected neural network. This method aims at learning the inverse mapping performed by two matrices used in the ISTA algorithm [22]. It is worth noting that this approach is computationally unfeasible for recovery large-size images such as high-resolution spectral images. On the other hand, ISTA-Net casts each ISTA iteration into a network structure that learns the soft-thresholding parameter, the regularization parameter, and a nonlinear transform [23]. Although the ISTA-Net structure is computationally efficient, this method is affected by the limited capacity of convolutional neural networks to learn the entire information embedded in the target image. Compared to ISTA-Net, the proposed approach unfolds a linearized version of the ADMM algorithm that alternately solves simple subproblems. Furthermore, our method attempts to learn a nonlinear transform that describes image details embedded in both the auxiliary variable and the Lagrange multiplier improving thus, the reconstruction performance while exploiting the computational efficiency of convolutional networks.
ADMM-CSNet unfolds the ADMM algorithm in the context of imaging compressive sensing [24,25]. In particular, this technique develops two matrix inversion procedures to estimate the target variable update, one of which is tailored to the measurement matrices used by compressive sensing magnetic resonance imaging (CS-MRI). In addition, ADMM-CSNet deploys a set of linear filters that attempts to learn the algorithm parameters, a linear transform, and the regularization function. Compared to the ADMM-CSNet, the proposed approach avoids resorting to matrix inversions by updating the target variable performing a computationally efficient proximal descent step. This minimization step can use any efficient measurement matrix such as random matrices and CSI sampling matrices. Furthermore, LADMM-Net aims at solving an ℓ 1 -regularized inverse problem whose proximal mapping reduces to the soft-thresholding operator on a shifted version of the auxiliary variable. Finally, our method optimizes an invertible nonlinear transform based on convolutional neural networks to sparsely represent image details. This paper is organized as follows. Section 2 describes the dual-arm architecture based on the 3D-CASSI systems, and Section 3 introduces the proposed approach for solving the image fusion problem from compressive projections. The results of extensive simulations are shown in Section 4, and some concluding remarks are exposed in Section 5.

Observational model
In this work, the proposed algorithm unfolding method is used to fuse spectral images from compressive measurements captured by a dual-arm acquisition system based on the 3D-CASSI architecture. To this end, we first introduce the foundations of the 3D-CASSI optical architecture, and then, we present the acquisition model of the dual-arm system.

3D-CASSI system
In general, the 3D-CASSI architecture is a compressive spectral imaging (CSI) system that has been designed to obtain the relevant information of spectral images by capturing a reduced number of camera snapshots [10]. This compressive acquisition system has been recently used in different applications such as clustering [30], feature fusion [31,32], spectral image classification [33], and spectral image fusion [14,15].
In essence, the 3D-CASSI system projects an encoded version of the input spectral image onto a focal plane array (FPA). A schematic of the 3D-CASSI optical architecture is illustrated in Fig. 1. As can be seen in this figure, f 0 (u 1 , u 2 , λ) stands for the spectral density source to be sensed, where (u 1 , u 2 ) represents the spatial location and λ stands the wavelength axis [11]. The input spectral field f 0 (u 1 , u 2 , λ) is firstly affected by the optical system as follows where h(u 1 , u 2 , λ) is commonly modeled as a shift-invariant impulse response that describes the propagation losses induced by the imaging optics. For the sake of simplicity, this impulse response has been typically modeled as h(u 1 , u 2 , λ) = δ(u 1 , u 2 , λ) [11]. Subsequently, f 1 (u 1 , u 2 , λ) is modulated by a colored coded aperture that consists of a 2D array of optical filters, where each optical filter at a specific spatial location modulates the incoming light according to a particular spectral response. It is worth noting that the encoding function depends on the pattern of optical filters forming the colored coded aperture [12]. More precisely, the encoding operation can be described as where t(u 1 , u 2 , λ) is the spatial-spectral encoding function performed by the colored coded aperture. Then, the encoded spectral field is integrated across the system spectral sensitivity Λ onto the FPA. In this case, the projected plane is given by To describe the intensity captured by an FPA element, consider ∆ the pixel width and rect u1 ∆ − i, u2 ∆ − j the surface covered by the FPA element at the discrete spatial location(i, j) [13]. Hence, the pixel intensity captured by the detector at the spatial coordinate (i, j) can be expressed as for i = 0, . . . , M − 1 and j = 0, . . . , N − 1, therefore, the detector has dimensions of M × N pixels.
A discrete model is frequently used to describe the 3D-CASSI compressive samples. To this end, consider the discrete data cube F ∈ R M×N ×L that describes the input spectral image whose elements at the spatial coordinate (i,j) and the ℓ-th spectral band are denoted as (F ) (i,j,ℓ) for i = 0, . . . , M − 1, j = 0, . . . , N − 1, and ℓ = 0, . . . , L − 1. Fig. 2 shows the discrete model of the optical phenomenon induced by the 3D-CASSI system to capture the compressive measurements. As can be observed in this figure, the encoding operation performed by the colored coded aperture is modeled as a binary data cube B ∈ {0, 1} M×N ×L with entries (B) (i,j,ℓ) whose values depend on the pattern of optical filters [30]. Afterward, the encoded image is obtained as the Hadamard product between the input spectral image F and the colored coded aperture model B, i.e.
F ⊙ B with ⊙ denoting the element-wise multiplication [12]. This encoded version is subsequently projected along the spectral axis onto a detector plane, and the measurement captured at the spatial location (i, j) can be obtained as where N ∈ R M×N is the noise matrix that affects the detector measurements whose entries (N) (i,j) are commonly assumed as independent and identically distributed (iid) random samples following a Gaussian distribution. In general, multiple snapshots are required to recover a reliable version of the spectral image from compressive measurements. In this regard, the 3D-CASSI sample obtained at the spatial coordinate (i, j) and the w-th snapshot can be defined as for w = 0, . . . , W −1, where B (w) is the model of the colored coded aperture used at the w-th snapshot, W the number of snapshots captured by the multi-frame 3D-CASSI system, and N (w) is the noise matrix affecting the w-th snapshot. Various methods that design the coded aperture patterns in the context of CASSI acquisition architectures have been developed in order to improve spectral image reconstructions [12,34].
In particular, we use the algorithm to design multi-frame colored-coded aperture patterns introduced in [31] that captures the entire spatial-spectral information of the input image without redundancy. Notice also that each 3D-CASSI snapshot captures an image with dimensions M × N pixels, therefore, the compression ratio between the captured data and the spectral image of interest is defined as η = MN W MN L = W L . Hence, a particular compression ratio is achieved by controlling the number of snapshots. Figure 3 illustrates a schematic of the dual-arm architecture that captures the multi-sensor compressive measurements. As can be seen in this figure, each arm consists of a 3D-CASSI optical system that projects an encoded version of the input spectral image onto the respective camera detector [10,35]. In particular, the upper part of the dual-arm system called MS 3D-CASSI includes a high-resolution coded aperture and a high-resolution camera detector. This arm captures poor spectral information due to the small sizes of the optical filters that form the colored coded aperture. More precisely, this arm aims at recovering a low-spectral-resolution multispectral image F ms = ζ(F ) with size M × N × L ms , where ζ(·) represents a spectral degradation function, and L ms = L/q denotes the number of bands of the multispectral image with q as the spectral decimation factor. Moreover, the colored coded aperture model used at the w-th snapshot can be described as B

Dual-arm acquisition system
. . , L ms − 1, and w = 0, . . . , W ms − 1, where W ms denotes the number of snapshots captured by the MS 3D-CASSI system. In consequence, a single sample captured by the MS 3D-CASSI system at the spatial coordinate (i,j) and w-th snapshot can be expressed as where N (w) ms ∈ R M×N is the noise matrix affecting the MS 3D-CASSI detector at the w-th snapshot. On the other hand, the bottom part of the dual system called HS 3D-CASSI consists of a low-spatialresolution coded aperture and a low-spatial-resolution camera detector. In contrast to the MS 3D-CASSI sensor, this arm captures rich spectral information of the input image. In this case, the HS 3D-CASSI sensor attempts to estimate the low-spatial-resolution image F hs = ξ(F ) with dimensions M hs × N hs × L, where ξ(·) represents the spatial downsampling function, and M hs = M/p; N hs = N/p with p as the spatial decimation factor. Furthermore, the coded aperture used at the w-th snapshot is modeled as B hs ) (i,j,ℓ) for i = 0, . . . , M hs − 1, j = 0, . . . , N hs − 1, and ℓ = 0, . . . , L − 1, and w = 0, . . . , W hs − 1, where W hs is the number of snapshots captured by the HS 3D-CASSI system. Therefore, a single measurement captured by the HS 3D-CASSI sensor at the spatial coordinate (i, j) and w-th snapshot can be defined as In summary, the entire set of compressive measurements obtained by dual-arm system can be succinctly described as y y y ms = H H H ms f f f + n n n ms , y y y hs = H H H hs f f f + n n n hs , where y y y ms ∈ R MN Wms and y y y hs ∈ R M hs N hs W hs are the vectors that contain the MS and HS compressive samples, respectively; H H H ms ∈ R MN Wms×MN L and H H H hs ∈ R M hs N hs W hs ×MN L are the measurement matrices describing MS 3D-CASSI and HS 3D-CASSI acquisition processes, respectively; f f f is the spectral image of interest in vector form; and n n n ms and n n n hs are noise vectors.
In particular, the measurement matrix of the MS 3D-CASSI sensor includes both the spectral decimation function ζ(·) and the effect of the colored coded aperture. In this case, the measurement matrix can be defined as for z m = 0, . . . , q. A representation of the measurement matrix H H H ms is illustrated in Fig. 4(a) for M = 8, N = 8, L = 4, q = 2, and W ms = 2. On the other hand, the measurement matrix of the HS 3D-CASSI system should consider the spatial downsampling function ξ(·) and the coded aperture pattern. More precisely, this measurement matrix can be expressed as for z h1 = 0, . . . , p and z h2 = 0, . . . , p. An example of the measurement matrix H H H hs is displayed in Fig. 4(b) for M = 8, N = 8, L = 4, p = 2, and W hs = 2.
Assuming that the noise vector entries are random samples obeying to a Gaussian statistical model, the image fusion problem reduces to minimize of the sum of the squared errors. However, this problem is ill-posed and its respective solution leads to severely degraded images. Hence, a regularization term is often included to exploit previous knowledge about the target spectral image. A sparsity-inducing term in a given transform transform is commonly included to solve this problem. Thus, the spectral image fusion problem from compressive measurements is formulated aŝ where λ 1 and λ 2 are regularization parameters.

Linearized ADMM
The alternating direction method of multipliers (ADMM) is a variable splitting optimization approach that has been widely used to solve inverse problems involving large-size datasets. This approach builds an augmented Lagrangian from a constrained optimization. Furthermore, the ADMM algorithm estimates the target variable by alternately solving small subproblems [29]. This optimization framework has successfully applied to solve compressive sensing problems with proven convergence properties [36,37]. In contrast to ISTA-based algorithms, ADMM-based approaches have shown in compressive sensing applications faster convergence rates and lower reconstruction errors [36]. In addition, in comparison with AMP-based methods, ADMM algorithms do not require tailoring a denoising engine to the application at hand. In the context of alternating direction methods, the spectral image fusion problem (13) can be reformulated as a constrained optimization given byf where b b b is an auxiliary variable under the ADMM framework. In this sense, the augmented Lagrangian associated with the constrained optimization (14) can be written as where ρ > 0 is a penalty parameter and d d d plays the role of the Lagrangian multiplier vector. The ADMM approach attempts to optimize the augmented Lagrangian by iteratively updating the target variable f f f , the auxiliary variable b b b, and the Lagrange multiplier vector d d d [29]. In this regard, the updating of the target variable at the iteration k is estimated by solving the minimization whose solution leads to a closed-form expression. However, notice that the estimation of f f f (k) involves computationally expensive matrix operations. To overcome this drawback, a linearized version of the ADMM algorithm obtains the first-order approximation of the cost function around f f f (k−1) [28] shown as follows where ∇(f f f (k−1) ) denotes the gradient of the cost function around f f f (k−1) given by By substituting (18) in (17), and minimizing the resulting cost function, the update f f f (k) can be obtained as In order to update the dual variable at the k-th iteration, the algorithm aims at solving the optimization that describes the proximal mapping when the regularizer is given by b b b 1 . Notice that the solution to this optimization problem is reduced to the soft-thresholding operation when the signal of interest is sparsely represented in a transform domain, i.e.

LADMM-Net
In this section, we introduce a deep learning architecture called LADMM-Net that exploits the advantages of both the LADMM optimization algorithm and the unrolling approach to solve the spectral image fusion problem from HS and MS compressive measurements. More precisely, the basic idea is to cast each iteration of the LADMM algorithm into a processing block with a convolutional neural network (CNN). Therefore, the overall architecture consists of multiple processing layers deploying a deep network. Fig. 5 shows a schematic of the proposed architecture for solving the spectral image fusion problem from HS and MS compressive measurements.
Notice that the data fusion problem attempts to describe the measurements that minimize the sparse image representation in a given transform domain. In general, combinations of orthogonal transforms such as the discrete cosine transform (DCT) or the orthogonal discrete wavelet transform (DWT) lead to degraded reconstructions. Therefore, an approach that includes nonlinear transform functions whose parameters can be learned from available data can be developed to improve the image recovery performance.
In this work, we follow a similar approach to that developed for the ISTA-Net method [23]. Basically, this approach substitutes the hand-crafted transform by a network structure that can be considered as a nonlinear transform function G(·) that induces sparsity to the image representation. In contrast to the ISTA-Net approach, the transform relied on CNN is optimized to estimate both the auxiliary variable and the Lagrange multiplier that capture the high-frequency information of the spectral image of interest.
Notice that the Lagrangian term encloses the information embedded in both the auxiliary variable and the Lagrange multiplier, therefore, the proposed network architecture attempts to optimize the image prior information and convexity-inducing penalty to improve the algorithm convergence. As shown below, the proposed approach exhibits a superior performance without resorting to the target variable mapping onto the residual domain. Figure 6 shows the structure of the network-based forward transform (NFT). As can be seen, this In addition, the proposed architecture also includes a network structure that recovers the image in its conventional domain. This structure, called network-based inverse transform (NIT), is also shown in Fig.   6. As can be seen in this figure, the network exhibits the same structure as the NFT network. It is worth noting that the NIT should be able to invert the effect of the forward transform, i.e.G(G(x)) ≈ x. This leads to architectures with end-to-end training schemes such that the whole network structure recovers the target image from compressive measurements. Furthermore, the symmetry of the network-based transform functions is considered during the training stage by including an invertibility error term in the loss function [23]. In this context, the spectral image fusion problem from HS and MS compressive measurements can be described asf An unrolled algorithm approach is developed to solve this problem that casts each iteration of the LADMM algorithm to a processing module dubbed LADMM-Net layer. This layer attempts to efficiently solve the updates (19), (21), and (22) by exploiting the flexibility of the network-based transform functions.
In essence, each LADMM-Net layer consists of an approximation unit (AU) and a network-based refinement unit (NRU) that are described as follows.

Approximation unit (AU)
This module estimate the target image according to (19). This unit is also shown in Fig. 5. Therefore, taking into account that f f f (k−1) and r r r (k−1) have been yielded by the previous layer, and considering the information embedded in the measurement matrices H H H ms and H H H hs , the output of this processing unit is expressed as where α (k) is a learnable step-size at the k-the phase; (λ (k) 1 , ρ (k) ) are the learnable regularization parameters at the k-the phase; and r r r (k−1) is the augmented Lagragian term computed by the NRU of the previous processing layer. Notice that, the learnable parameters (α (k) ,λ

Network-based refinement unit (NRU)
An schematic of this module is also illustrated in Fig. 5. In particular, the NRU aims at obtaining the update of the dual variable b b b (k) and the Lagrange multiplier d d d (k) . Taking into account the fusion problem described in (23), the dual variable estimate aims to solve the optimization problem given bŷ whose solution is obtained asb whereλ (k) = λ 2 /ρ is the soft-thresholding parameter that is learned at the k-th processing stage. To obtain the dual variable update b b b (k) , the current estimate of the target variable f f f (k) should be sparsely represented by the nonlinear transform function G(·) recreated by the convolutional neural network. This transform performed by the NFT block can be described asf f f (k) = G(f f f (k) ). Then, the dual variable update is obtained according to (26). In other words, the soft-thresholding is applied to a shifted version of the image transform.
The Lagrange multiplier vector estimated by the previous layer d d d (k−1) should be available.
Subsequently, the Lagrange multiplier update is obtained as that also requires d d d (k−1) . Therefore, as can be seen in Fig. 5, the dual variable update d d d (k) is defined as a layer output.
Upon a closer look of (19), the term ) can be seen as the contribution of the augmented Lagragian term. In particular, this vector contains the information of the coefficients removed by the soft-thresholding function that commonly contains both the high-frequency components of the image of interest and coefficients associated with the measurement noise. As can be seen in (24), the terms to compute the target variable update are available. Therefore, the residual is computed in the transformed domain and then the network-based inverse transform (NIT) is applied to this subtraction, in other words, where r r r (k) is also defined as an output of the processing layer. The information of the dual variable is implicitly included in this residual vector. Hence, the proposed architecture considers the high-frequency information of the spectral image without remarkably increasing the memory requirements.

LADMM-Net parameters
Basically, the proposed architecture attempts to learn a set of parameters denoted by Θ whose elements are described as follows. First, the step-size α (k) and the regularization parameters (ρ (k) ,λ 1 ) are updated in every approximation unit. Furthermore, the thresholdλ (k) and the parameters of the network-based nonlinear transforms G (k) andG (k) are learned in the NRU. In summary, the proposed deep network can be described by the set of parameters Θ where K is the number of the LADMM-Net layers. The number of learnable parameters that contains the proposed architecture is K(4 + 36M L).

Initialization
An initial estimate f f f

LADMM-Net training
In the context of the spectral image fusion from HS and MS compressive measurements, the ground truth image is denoted by f f f v . On the other hand, the corresponding HS and MS compressive projections (y y y ms , y y y hs ) v can be described as the input dataset. Therefore, the training set can be built as Γ = {f f f v , (y y y ms , y y y hs ) v } B b=1 . In this work, we use a similar loss function than that reported in [23] for the ISTA-Net. Specifically, the training stage attempts to minimize the mean square error H(Θ) data while an invertibility error H(Θ) inv that estimates the symmetry degree of network-based nonlinear transform functions is also considered. Thus, given the training set Γ, the loss function is defined as with where f f f v (Θ, Γ) is the fused image recovered by the proposed architecture using the set of learnable parameters Θ and the compressive measurements (y y y ms , y y y hs ) v . Note that K is the number of LADMM-Net layers and B is the batch size. In this work, γ is set to 0.1.
As can be seen, the proposed approach unfolds the linearized ADMM algorithm whose convergence analysis is developed in [36,37]. In contrast to the linearized ADMM algorithm that uses a predefined transform, the proposed approach attempts to learn a nonlinear transform describing the image detail information embedded in both the dual variable and the Lagrange multiplier. In this regard, the convergence of the proposed approach shall also depend on the parameter setting at the training stage, including the initialization of the learnable weights and the learning rate. Therefore, since the nonlinear nature of the learned transform and the huge amount of parameters involved at the learning stage, the convergence analysis is considered an open research problem.

Results
To evaluate the proposed spectral image fusion approach, we use two available databases: the Harvard database and the CAVE database. Furthermore, we test the proposed approach to fuse a spectral image from HS and MS compressive measurements captured at the laboratory. Finally, the proposed algorithm unrolling technique is extended to recover grayscale images from compressive random projections.

Harvard database
This database consists of fifty high-resolution images that exhibit indoor and outdoor scenes under natural illumination [38]. Since the proposed approach is compared to other model-based methods that sparsely represent the spectral image in a predefined transform domain, we extracted 72 spectral images with dimensions 512 × 512 pixels and 31 spectral bands in the wavelength range from 420 nm to 720 nm.   Table 1 shows the averages of various performance metrics over 24 testing images as the number of layers increases. More precisely, we include the ensemble average of the PSNR, SSIM, the spectral angle mapper (SAM), the running time, and the training time. As can be observed in this table, the performance of the proposed approach improves as the number of processing layers increases. It can be also seen that the computation times are larger as the number of layers increases. Nevertheless, the proposed method yields a remarkable performance for a small number of processing blocks. It is worth noting that the spectral image fusion involves large data sizes, and multiple variables with the target image size should be stored by each processing layer. Therefore, the maximum number of processing layers is constrained by the GPU memory size.
For comparison purposes, different methods that fuse spectral images from compressive measurements are implemented. The first approach aims at recovering both the HS image and the MS image from their respective compressive measurements, and then, the fast fusion based on Sylvester equation (FUSE) [8] is applied to obtain the high-resolution spectral image. This approach is referred to as Rec + FUSE. In this case, an ADMM-based algorithm that solves an ℓ 1 -regularized inverse problem is used to reconstruct both the HS image and the MS image. Secondly, the spectral image fusion from compressive measurements (SIFCM) method [14] is implemented. Additionally, ISTA-Net [23] and OPINE-Net methods [39] are adapted to the spectral image fusion problem from compressive measurements. In this experiment, LADMM-Net, ISTA-Net, and OPINE-Net are implemented using 10 processing layers and 48 training samples. For these deep learning architectures, we use the Adam optimization algorithm and a learning rate ζ = 0.0005. Fig. 8 displays the RGB composite of the reconstructed images using a compression ratio of 25%. Notice that the PSNR values are included in the bottom part of every recovered image. In addition, Fig. 9 shows the spectral signatures of selected pixels belonging to the recovered images. As can be observed in these figures, the proposed architecture achieves the best performance with respect to the other approaches.

CAVE database
In this experiment, we use the Stuff section of the CAVE spectral image database. This database contains fifteen high-resolution images of previously constructed scenes under controlled illumination [40].
Furthermore, the spectral images exhibit dimensions of 512 × 512 × 31 that include reflectance data from 400 nm to 700 nm. The first column of Fig. 10 Fig. 10 shows the RGB composite of spectral images recovered by the various methods. In this case, LADMM-Net, ISTA-Net, and OPINE-Net are implemented with 10 processing layers. As can be observed, the proposed image fusion method outperforms the remaining methods. Table 3 displays the performance results obtained by the different spectral fusion methods in terms of PSNR, SSIM, SAM, running time, and training times. As can be seen in this table, the proposed approach outperforms the other methods in terms of image quality metrics, while it also exhibits competitive computation times.
We also evaluate the performance of the proposed fusion approach using a different acquisition setting.
More precisely, the dual-arm acquisition architecture is simulated to obtain a single monochrome snapshot instead of multiple MS 3D-CASSI snapshots. In other words, the MS 3D-CASSI arm captures a single    high-spatial-resolution grayscale image with a 31:1 spectral decimation factor (q = 31). On the other hand, the HS 3D-CASSI acquires multi-frame compressive snapshots with a 16:1 spatial downsampling factor (p = 4). This case can be considered as a compressive hyperspectral pansharpening that attempts to obtain a high-resolution spectral image from hyperspectral compressive measurements and a high-spatial-resolution panchromatic image. The first column of Fig. 11 displays HS 3D-CASSI snapshots, while the second column shows high-spatial-resolution grayscale images. The last four columns in Fig. 11 show the RGB composites obtained by various fusion methods for this particular acquisition setting. As can be observed in this figure, the proposed architecture outperforms the other approaches achieving an improvement gain of at least 3 dB.

Laboratory data set
We also evaluate the proposed reconstruction approach from compressive measurements captured by an optical setup that implements the dual-arm system based on the 3D-CASSI architecture. To be more precise, the experimental setup comprises a 100 mm objective lens, a 100 mm relay lens, a digital micromirror device (DMD), and a camera detector based on a charged coupled device (CCD). To obtain the dual-resolution compressive measurements the patterns of the coded aperture were separately emulated using the DMD. In particular, we fixed the spatial resolution of the DMD to 256 × 256 to capture the compressive measurements of the multispectral arm, while the DMD spatial resolution was fixed to 128 × 128 to obtain the compressive

CS reconstruction of natural images
The proposed algorithm unrolling technique can be adapted to solve a particular compressive sensing problem. In this section, the performance of the proposed architecture is tested to recover grayscale images from compressive random projections. In this regard, the target variable update is given by where the CS-based acquisition process can be described as y y y = H H Hf f f + η η η, where y y y ∈ R m is the vector containing the compressive measurements; H H H ∈ R m×n denotes the measurement matrix with m ≪ n; f f f ∈ R n is the image of interest in vector form; and η η η ∈ R M is the noise vector whose entries are frequently characterized as iid random samples following a zero-mean Gaussian distribution [41,42]. Notice that this network architecture can be used in other compressive sampling applications including sparse channel estimation [43], smart grids [44], among others.
In this sense, we obtain the training data by extracting B = 5120 image blocks with dimensions 33 × 33 from the Berkeley BSD dataset [45]. Furthermore, the testing data set includes 12 standard images with dimensions 256 × 256. For a given compression ratio, the measurement matrix is generated whose elements are random samples following a zero-mean Gaussian distribution. The sampling matrix columns {h  } B

=1
are normalized such that h  2 2 = 1 for  = 1, . . . , B. We compare the performance of the proposed deep network with respect to other reconstruction techniques such as the gradient projection for sparse reconstruction (GPSR) [46], the linearized version of the alternating direction method of multipliers (LADMM) [28], the denoising-based approximated message passing (D-AMP) method [47], the ReconNet architecture [48], the LISTA approach [22]  (Right) PSNR obtained by the proposed approach versus the epochs for different compression rates.  [49], the ISTA-Net network [23], and the OPINE-Net architecture [39]. Notice that GPSR, LADMM, and D- To evaluate the performance of the proposed algorithm unrolling approach at the training stage, Fig.   13(left) displays the average PSNR of the images recovered by the various network-based methods as the number of learning epochs increases. As can be seen in this figure, the proposed approach exhibits an outstanding performance for the entire evaluation interval. Moreover, Fig. 13  To quantitatively evaluate the proposed technique, Table 4 shows the ensemble average of the PSNR in dB obtained by the various CS reconstruction methods from compressive random projections for different compression ratios. The higher values are in bold font. As can be seen in this table, the proposed approach exhibits a remarkable outcomes across the compression ratios under test. In addition, Fig. 14 illustrates the images recovered by the various CS reconstruction methods from compressive measurements with a compression ratio of 25%. As can be observed in this figure, the proposed approach is able to properly recover smooth-wise regions preserving, in turn, the structure of the edges. Finally, Fig. 15 displays the reconstructed image obtained by various methods relied on deep netwoks. Fig. 15 displays the recovered images obtained by the different reconstruction methods relied on deep networks. Outstanding performance is exhibited by the proposed approach.

Conclusion
In this work, a deep learning architecture under the algorithm unrolling approach was proposed for estimating high-resolution spectral images from multi-sensor compressive measurements. More precisely, the   proposed image fusion method cast each iteration of a linearized version of the ADMM algorithm into a CNNbased structure whose concatenation of deploys a deep network. The linearized approach allows estimating the target variable without resorting to computationally costly matrix inversions. Therefore, the proposed image fusion architecture exhibits competitive running times. On the other hand, the network-based structure learns the relevant information embedded in both the auxiliary variable and the Lagrange multiplier that improves the estimation performance. Notice that the network-based architecture learns its parameters using an end-to-end training scheme. In addition, we evaluated the performance of the proposed approach on two spectral image databases and one real data set. The results yielded by the proposed architecture outperformed those obtained by state-of-the-art methods. To since the proposed approach can be seen as a deep unrolling method for solving inverse problems, the proposed architecture was tested to reconstruct natural images from compressive random projections. In future work, we are interested in evaluating other regularization functions and training different deep networks such as convolutional generators, variational autoencoders, and generative adversarial networks. Furthermore, we are also interested in implementing the proposed approach in further imaging applications including image denoising, inpainting, and deblurring.

Acknowledgments
This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sk lodowska-Curie grant agreement No 754382, GOT ENERGY TALENT. The content of this article does not reflect the official opinion of the European Union. Responsibility for the information and views expressed herein lies entirely with the authors.