{'fraction_non_alphanumeric': 0.03913917940466613, 'fraction_numerical': 0.013716814159292035, 'mean_word_length': 4.651511707206184, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ConvCSNet: A Convolutional Compressive Sensing Framework Based on Deep Learning

Compressive sensing (CS), aiming to reconstruct an image/signal from a small set of random measurements has attracted considerable attentions in recent years. Due to the high dimensionality of images, previous CS methods mainly work on image blocks to avoid the huge requirements of memory and computation, i.e., image blocks are measured with Gaussian random matrices, and the whole images are recovered from the reconstructed image blocks. Though efficient, such methods suffer from serious blocking artifacts. In this paper, we propose a convolutional CS framework that senses the whole image using a set of convolutional filters. Instead of reconstructing individual blocks, the whole image is reconstructed from the linear convolutional measurements. Specifically, the convolutional CS is implemented based on a convolutional neural network (CNN), which performs both the convolutional CS and nonlinear reconstruction. Through end-to-end training, the sensing filters and the reconstruction network can be jointly optimized. To facilitate the design of the CS reconstruction network, a novel two-branch CNN inspired from a sparsity-based CS reconstruction model is developed. Experimental results show that the proposed method substantially outperforms previous state-of-the-art CS methods in term of both PSNR and visual quality.


I. INTRODUCTION
Compressive sensing (CS) is a new image/signal acquisition framework acquiring only a few linear measurements [1], [2] with random measurement matrices. Such framework has the potentials of significantly improving the imaging speed and sensor energy efficiency in real applications. Based on CS, several new imaging systems, including single-pixel camera [3], compressive spectral imaging system [4], high-speed video camera [5], and fast Magnetic Resonance Imaging (MRI) system [6] have been developed. Perfect reconstruction from the linear measurements is guaranteed if the following two conditions are met, i.e., the sensing matrices have the restricted isometry property (RIP) and the images have sparse representation with respect to a dictionary. These two conditions can be easily satisfied as random Gaussian matrices have the RIP with high probabilities [1], [2] and natural images have sparse representation under many off-the-shelf or learned dictionaries (e.g., wavelets, learned dictionaries [7]). However, in practice the promise of CS is often offset by challenges relating to two conditions. First, it is difficult or even impossible to implement the random sensing matrix for a large image. Second, the sparsity-based CS reconstruction algorithms are very slow to converge for obtaining good estimates of the original images.
X. Lu, W. Dong, P. Wang, G. Shi, and X. Xie are with School of Electronic Engineering, Xidian University, Xi'an, 710071, China (e-mail: wsdong@mail.xidian.edu.cn) For a given image x ∈ R N , the CS measurement of x can be expressed as y = Φ, where Φ ∈ R M ×N is the sensing matrix and M = rN (0 < r 1). In practice, as the dimension of natural images are often very high, the memory requirement for storing the matrix Φ is huge. For example, for a moderate image of size 1, 000 × 1, 000 and r = 0.1, the dimension of Φ is 100, 000 × 1, 000, 000. The computational complexity of the CS reconstruction with such large sensing matrix is also prohibitively high. To avoid these difficulties, block-based CS (BCS) methods have been proposed [8], [9], where an image is divided into many nonoverlapping blocks and each block is sensed individually. As such, the sensing matrices Φ become much smaller and images can be efficiently measured. However, as the blocks are sensed and reconstructed individually, the BCS methods lead to serious blocking artifacts. To reduce the blocking artifacts, post-processing is often required to improve the visual quality.
According to the CS theory [1], [2], the original images can be well-reconstructed by exploiting the sparsity prior of natural images. However, the sparsity-based CS methods recover the original images by solving an optimization algorithm, which is very slow to converge. Thus, the optimization-based CS methods cannot be used for real-time applications. Recently, inspired by the great successes of deep neural network (DNN) for computer vision tasks [10]- [12], DNN-based CS reconstruction methods have also been proposed [13]- [15]. With DNN, both the linear sensing and nonlinear reconstruction can be performed by a single neural network. Through end-to-end training, both the sensing matrix and the reconstruction method can be jointly optimized, and the speed is usually hundredstimes faster than optimization-based methods. However, to the best of our knowledge, current DNN based methods are all block-based, i.e., image blocks are sensed and reconstructed individually. Blocking artifacts can also be observed in the reconstructed images by the DNN-based methods [13], [14].
In this paper, we propose a novel convolutional compressive sensing (ConvCS) framework based on deep convolutional neural network (DCNN). In the proposed ConvCS network, the first layer senses an input image by convolving the whole image with a set of random filters, followed by subsampling. The advantage of the proposed ConvCS is that the whole image can be efficiently sensed with a set of small filters that are easy to store, and effectively reconstructed without introducing blocking artifacts. The remaining layers of the proposed ConvCS network perform the nonlinear reconstruction of the whole image from the measurements. To design the reconstruction network, the domain knowledge of the sparsity-based CS reconstruction is incorporated, leading to a novel CNN for image reconstruction. By end-to-end training, both the convolutional sensing filters and the reconstruction CNN are jointly optimized. Experimental results show that the proposed method substantially outperforms current stateof-the-art CS methods in terms of PSNR and visual quality.

II. RELATED WORKS A. Background of CS
Instead of sampling the entire signal, the CS theory samples a signal x ∈ R N by taking only M linear measurements, i.e., y = Φx + e, y ∈ R M , where Φ ∈ R M ×N , M < N is the sensing matrix and e ∈ R M is the measurement noise. Since M < N , recovering x from y is generally an ill-posed inverse problem. However, the CS theory guarantees perfect reconstruction of x if x is sparse in some sparsifying spaces and Φ holds the RIP. It has been proven that the Gaussian random matrices have the RIP with very high probabilities. Standard CS methods recover x by solving a minimization problem, where D is the dictionary and α are the sparse codes of x.
After estimating the sparse codes α, x can be reconstructed as x = Dα. It has been proven in [1], [2] that x can be faithfully recovered from M = O(K log(N/K)) measurements, where K denotes the number of nonzero coefficients of α. The 1minimization problem can be solved by many optimization algorithms [16], [17]. However, as mentioned above these methods are very slow to converge.

B. Block-based image CS
When applying CS to images, the image can be sensed by representing it into a vector and measured as y = Φx. However, for images of high dimensionality, the sensing matrix Φ becomes very large and it is impossible to store it and compute with such large matrix Φ. To avoid such difficulties, block-based CS (BCS) methods have been proposed [8], [9]. In these methods, the input image is divided into many nonoverlapped blocks and each block is sensed independently using a much smaller matrix Φ. The full image is recovered by placing back the reconstructed blocks, followed by fullimage smoothing. To reduce the blocking artifacts, full-image iterative shrinkage algorithm was proposed [9], and improvements can be achieved by using more advanced transforms, such as contourlets and dual-tree discrete wavelet transforms [8]. In addition to the sparsity prior, model-based CS recovery algorithms have also been developed to exploit the high-order dependencies between the wavelet coefficients [18], leading to better performance. The effective nonlocal self-similarity prior has also been integrated into the objective function through nonlocal low-rank regularization [19].

C. Structured CS for images
To overcome the drawback of the BCS, structured CS operators [20]- [22] have been proposed. The convolutional CS methods, which performs the sensing by first convoluting a signal with a random filter and then subsampling, have been proposed in [20], [22]. These CS methods are easy to implement and have many potential applications, such as Fourier optics [20], Radar imaging [21] and coded aperture imaging [23]. In addition to the random filter, deterministic filter that is more convenient to implement has also been proposed in [22]. Though the convolution-based CS are more easy to implement, the use of only one random filter makes the reconstruction problem more difficult [20]. Also, all these convolution-based CS use iterative algorithms to reconstruct the original images, which are very slow to converge.

D. Deep learning based CS for images
Recently, inspired by the successes of the deep neural networks, non-iterative CS reconstruction methods have been proposed [13]- [15]. In [13], a stacked auto-encoder denoising network was developed for image CS. Similarly, a fullconnected neural network has been proposed in [15]. Convolutional neural network has also been proposed for this task, where a BM3D denoising [24] stage is adopted to further improve the reconstruction performance [14]. Through end-toend training, both the sensing matrices and the reconstruction network can be jointly optimized for better performance. However, all those methods perform CS and reconstruction on image blocks, leading to limited reconstruction performance.
In this paper, we propose a new convolutional compressive sensing (denoted as ConvCS) framework using deep convolutional neural network (DCNN), where the first layer implements the sensing by convolving the input image with a set of random filters followed by subsampling. The remaining layers reconstruct the input image from the linear measurements. Furthermore, inspired from the sparsity-based reconstruction model, a novel CNN containing two branches is proposed for CS reconstruction. By performing sensing and reconstruction on the whole image, the proposed method significantly outperforms the previous CS methods. Different from previous convolution-based CS methods [20], [22], both the sensing filters and the reconstruction algorithm can be jointly optimized. Experimental results show that the proposed method outperforms existing state-of-the-art CS methods by a large margin.

III. PROPOSED CONVOLUTIONAL COMPRESSIVE SENSING USING DEEP LEARNING A. Proposed convolutional compressive sensing
Unlike existing BCS and convolutional CS, we propose to sense an image by convolving it with a set of random filters, followed by spatial subsampling of the convolved images. Specifically, for a given image x ∈ R N , we convolve it with a set of random filters φ i of size L × L, i = 1, 2, · · · , m, to generate the CS measurements. Mathematically, the sensing matrix can be expressed as where Φ i ∈ R N ×N is set to be a sparse matrix such that Φ i x is equivalent to convolving x with filter φ i , and S ∈ R M0×N is a subsampling matrix. Then, the proposed convolutional CS (ConvCS) for x can be formulated as y = Φx, y ∈ R M , where M = mM 0 . As convolution can also be implemented with matrix-vector multiplication, the proposed ConvCS is equivalent to the CS process that first extracts blocks of size L×L with sliding step s and then measures the blocks with the sensing matrix, whose i-th rows are composed with the vectorized filter coefficients of φ i . Thus, the proposed ConvCS matrix Φ of Eq. (2) still holds the advantages of the random Gaussian matrix for CS (i.e., the RIP). However, the proposed ConvCS is clearly distinct from the previous BCS methods in two aspects. First, the proposed ConvCS for images of large dimensions is much easier to implement. Second, the convolutional nature of the proposed ConvCS makes the joint optimization of the sensing filters φ i and reconstruction of the whole image much more effective, without introducing any blocking artifacts.
The proposed ConvCS can be easily implemented using a convolutional neural network, as shown in Fig. 1 (a). The first layer convolves the input image x with a set of m random filters of size L × L with stride s. The measurements can be obtained by representing the obtained feature maps into a Mdimensional vector y. Note that nonlinearity is not involved in the CS process. More convolutional layers with nonlinear activity function could be added into the ConvCS process, which may lead to better performance. However, for simplicity, here we only use one linear layer to obtain the measurements. As shown in Sec. IV, the simple ConvCS can already lead to excellent CS recovery performances.
Essentially, the ConvCS encodes the visual information of images and plays a similar role as autoencoders. In the context of CS, the stacked denoising autoencoders (SDA) [13] have been proposed to encode the images. However, the SDA method performs CS encoding at image block levels, resulting in serious blocking artifacts.

B. Sparse model inspired DCNN for CS reconstruction
After obtaining the CS measurements, we aim to recover the original image from the measurements. In recent years, DCNN has shown very promising performances for many low-level image processing tasks, e.g., image super-resolution [25], [26]. However, the problem of reconstructing images from the ConvCS measurements is different from previous image SR problem, and existing reconstruction network can not be applied directly for this task. To facilitate the design of the CS reconstruction network, we propose to incorporate the domain knowledge of the sparsity-based CS reconstruction. Specifically, we first propose the following analysis sparse representation model for CS reconstruction, where w k ∈ R n×n is the analysis filter, * denotes the 2D convolution, and J(·) denotes a regularization term imposed on sparse codes α k . Classic sparsity enforcing regularizers, e.g., || · || p (0 ≤ p ≤ 1), as well as the non-negative indicator function I(·) (as inspired by the ReLU function) can be adopted. Solving Eq. (3) amounts to alternatively solving two subproblems, i.e., where k = 1, 2, · · · , K. Both sub-problems can be easily optimized. With a fixed estimate x (t) obtained at t-th iteration, α k can be solved in closed-form for several commonly used sparsity regularizer, as for 1 -norm sparse regularizer and non-negative regularizer, respectively, where S τ (·) denotes the soft-thresholding with threshold τ . The x-subproblem is a quadratic optimization problem and can be solved in closed-form. However, to avoid large matrix inversion, we prefer to solve it via a gradient descent method. With a fixed estimate of α k , we iteratively update x as where W k is the sparse matrix such that W k x is equivalent to convolving x with w k , and δ is the predefined constant. By alternatively updating x and α, the iterative process will converge. However, the convergence speed is very slow.
In this paper, we propose to convert the alternative update of x and α into a deep network. For simplicity, we rewrite Eq. (6) as where Λx = Φ Φx denotes the process that first senses x with Φ and then reconstructs x from the CS measurements by back-projection. For simplicity, we let Λ = I and W k W k = I. Then, Eq. (7) can be approximated as where ρ = (1 − δ(1 + η)), γ = δη, x (0) denotes the initial reconstruction of x from y, and . Inspired from the alternative update of α k and x, we propose a novel DCNN for CS reconstruction. As shown in Fig. 1 (b), the proposed reconstruction network contains two branches. The first branch implements a conventional CNN for generating the feature maps α k . The first branch taking the measurements vector y as input back projects y into the feature maps, where all entries are zero except the sampled set of pixels (as marked red shown in Fig. 1 (b)). The first layer uses kernels of size L × L and generates m feature maps, while the remaining 14 layers use kernels of size 3 × 3 and generate 96 feature maps. The ReLU function is applied following convolution. Compared to the directly implementation of Eq. (5), the deep CNN is more powerful in learning the representation of the original image x. The first CNN branch can also be regarded as a nonlinear mapping function used to accurately predict the sparse codes α k . The second branch recursively reconstructs the image based on the feature maps α from the first branch and the previously reconstructed images x (t) and x (0) , which mimics the computations of Eqs. (7) and (8). In each layer of the second branch, the feature map α (t+1) from the CNN branch is fed into a convolutional layer to produce an image, which is further added with previously reconstructed x (t) and x (0) for an updated estimate x (t+1) . The kernel size used in the reconstruction branch is also 3 × 3.

A. Training details
To achieve better performance, the sensing layers and the remaining reconstruction layers are jointly trained, and thus the sensing filters φ i and the reconstruction network can be jointly optimized. Let ConvCSNet denote the proposed network performing convolutional CS and reconstruction. To verify the effectiveness of the proposed two-branch reconstruction network, we also implement a variant of the ConvCSNet (denoted as ConvCSNet-baseline), which uses only the first branch of the reconstruction network shown in Fig. 2 (b) for CS reconstruction. To train the proposed networks, we collected natural images from the ImageNet dataset [27] and extracted the central 160 × 160 part of each image. The extracted patches are converted into grayscale and augmented via horizontal and vertical flips and 90 rotations. Finally, we obtained a training set containing 4000, 000 image patches. We empirically set the parameters of the convolutional sensing layer as: L = 17, m = 8, s = 8 for rate 0.05; L = 17, m = 6, s = 8 for rate 0.1, L = 11, m = 5, s = 5 for rate 0.2, and L = 11, m = 8, s = 5 for rate 0.3, where L, m and s denotes the filter size, number of filters and the convolutional stride, respectively. The parameters of the reconstruction layers are the same for different measurement rates, except the first layer of the reconstruction part of the ConvCSNet. In that layer, we use corresponding filter size of L×L used in the sensing layer.
The proposed network is trained using the 2 loss function, as L(Θ) = 1 the network parameters (including the sensing filters) and T is the total number of training patches. The proposed network is trained with ADAM optimizer [28] by setting β 1 = 0.9, β 2 = 0.999 and = 10 −8 . The minibatch size is set to 64.
We initialize the learning rate as 10 −4 and halved at every 2 × 10 5 minibatch updates. We implemented the proposed network with TensorFlow framework and training them using 4 NVIDIA 1080Ti GPUs. It takes one day to train ConvCSNet. Currently, we trained one model for each sensing rates. In future, a general reconstruction network may be trained for different measurement rates.

B. Comparison with state-of-the-art methods
We compare the proposed ConvCSNet method with two iterative CS methods, including the total variation method (denoted as TV [29]), the denoising-based approximate message passing (D-AMP) method [30], and one recently developed fully-connected network based CS method (denoted as FCN-CS) [15]. Note that both TV [29] and D-AMP methods [30] conduct compressive sensing on the whole image, using very large measurement matrices. For an input image of size 255 × 255 and measurement rate 0.2, the measurement matrix is of size 13005 × 65032, requiring about 7GB memory for storing it in Matlab platform. The computational complexity of the iterative CS reconstruction using such large sensing matrices also become very slow. However, the advantage of using such large measurement matrices is that the reconstruction quality can be much improved. Also note that the D-AMP method [30] uses the well-known BM3D denoising method [24] in its iterative reconstruction process. As the BM3D denoising method is very effective in suppressing noise and artifacts, the D-AMP method achieves the state-of-the-art CS performance. The deep learning based FCN-CS method [15] performing the sensing and reconstruction based on 16 × 16 blocks. As the authors of FCN-CS method only provided the test code in their website, we re-implemented the training algorithm of FCN-CS method and trained the network with our training dataset. As we use larger training dataset, the performance of FCN-CS method is much improved, compared with those obtained using the model provided by the authors of [15]. We have also tried to compare our method with the Re-conNet of [14]. However, the results obtained using the code downloaded from their website are worse than those reported in their paper. We think that this may be caused by different parameters settings, and tuning the parameters of the method for better results is out of the scope of this paper. Hence, we didn't include the ReconNet [14] into our comparison study. All the codes of the competing methods are downloaded from authors' websites. We generate the measurements at four measurement rates 0.05, 0.1, 0.2, and 0.3. To verify the robustness of the reconstruction algorithms to the measurement noise, we also conducted CS reconstruction using noisy measurements. To this end, Gaussian noise of standard deviation of 10 is added to the measurements. A set of natural images of size 255×255 is used as test images, as shown in Fig. 2. The well-known Berkeley segmentation dataset containing 100 natural images (denoted as BSD100) is also used to verify the performances of the test methods. For the BSD100 dataset, we extract the central 320 × 320 part of each image as test images. Note that all the test images are excluded from the training dataset. Table I reports the PSNR results of the test methods on the set of test images shown in Fig. 2. It can be seen that the proposed ConvCSNet outperforms the ConvCSNet-baseline method by a large margin, demonstrating the effectiveness of the proposed CS reconstruction network. When compared with other competing methods, the proposed ConvCSNet method also performs better than the other methods on all measurement rates. It significantly outperforms the block-based FCN-CS method on all measurement rates. The D-AMP method performs very well for high measurement rates (e.g., 0.2 and 0.3). The proposed ConvCSNet also performs much better than D-AMP method on all measurement rates. For noisy cases, the proposed ConvCSNet also significantly outperforms the other methods by large margins.    Table III. For TV [29] and D-AMP [30] methods, the computer with Intel i7-6700 3.4GHz and 64G memory is used to run these algorithms provided by the authors. For the FCN-CS [15] and the proposed methods, a Nvidia GTX 1080Ti GPU is used to compute the CS reconstructions. From Table III, it can be seen that the time taken by the proposed ConvCSNet for reconstructing a 255×255 image is about 1030 times faster than the TV method [29] and 760 times faster than the D-AMP method [30]. Note that the high computational complexity of TV and D-AMP is not only due to slow convergence, but also the use of huge measurement matrices. When compared to the block-based FCN-CS method, our method is also about 5 times faster.
V. CONCLUSIONS Compressive sensing is a new image/signal acquisition paradigm, which has potentials in high-speed and energy efficient imaging applications. However, a practical issue in the use of CS theory is the huge memory and computation requirements for sensing a whole image. While performing CS on image block level leads to efficient measurement, it will significantly decrease the reconstruction performance. In this paper, we propose a novel convolutional compressive sensing (ConvCS) method based on deep learning. In the proposed ConvCS network, the first layer conducts sensing of the whole image using a set of convolutional filters and the remaining layers performs the reconstruction of the whole image. For better CS reconstruction, a novel two-branch convolutional neural network is proposed. Through end-to-end training, both the sensing filters and the reconstruction network can be jointly optimized. Experimental results show that the proposed method significantly outperforms existing iterative and deep learning based CS methods.