{'fraction_non_alphanumeric': 0.04033279073973594, 'fraction_numerical': 0.012479652740097666, 'mean_word_length': 4.673301867432793, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dual Reconstruction Nets for Image Super-Resolution with Gradient Sensitive Loss

Deep neural networks have exhibited promising performance in image super-resolution (SR) due to the power in learning the non-linear mapping from low-resolution (LR) images to high-resolution (HR) images. However, most deep learning methods employ feed-forward architectures, and thus the dependencies between LR and HR images are not fully exploited, leading to limited learning performance. Moreover, most deep learning based SR methods apply the pixel-wise reconstruction error as the loss, which, however, may fail to capture high-frequency information and produce perceptually unsatisfying results, whilst the recent perceptual loss relies on some pre-trained deep model and they may not generalize well. In this paper, we introduce a mask to separate the image into low- and high-frequency parts based on image gradient magnitude, and then devise a gradient sensitive loss to well capture the structures in the image without sacrificing the recovery of low-frequency content. Moreover, by investigating the duality in SR, we develop a dual reconstruction network (DRN) to improve the SR performance. We provide theoretical analysis on the generalization performance of our method and demonstrate its effectiveness and superiority with thorough experiments.


Introduction
Super-resolution (SR) aims to learn a nonlinear mapping to reconstruct high-resolution (HR) images from low-resolution (LR) input images, and it has been widely desired in many real-world scenarios, including image/video reconstruction [8,17,22,38], fluorescence microscopy [35] and face recognition [7]. SR is a typical ill-posed inverse problem. In the last two decades, many attempts have been made to address it, mainly including interpolation based methods [48] and reconstruction based methods [8,12,17,19,21,25].
Recently, deep neural networks (DNNs) have emerged as a powerful tool for SR [8] and have shown significant advantages over traditional methods in terms of performance and inference speed [8,24,25,28]. However, these methods may have some underlying limitations. First, for deep learning based SR methods, the performance highly depends on the choice of the loss function [25]. The most widely applied loss is the pixel-wise error between the recovered HR image and the ground truth image, such as the mean squared error (MSE) and mean absolute error (MAE). This kind of losses is helpful to improve the peak signal-to-noise (PSNR), a common measure to evaluate SR algorithms. However, they may make the model lose the high-frequency details and thus fail to capture perceptually relevant differences [20,25]. To address this issue, recently, some researchers have developed the perceptual loss [20,25] to produce photo-realistic images. However, the computation of perceptual loss relies on some pre-trained model (such as VGG [39]), which has profound influence on the performance. In particular, the method may not generalize well if the pre-trained model is not well trained. In practice, it may incur significant changes in image content (see Figure 2 in [25] and Figure 3 in this paper).
Moreover, most deep learning methods are trained in a simple feed-forward scheme and do not fully exploit the mutual dependencies between low-and high-resolution images [13]. To improve the performance, one may increase the depth or width of the networks, which, however, may incur more memory consumption and computation cost, and require more data for training. To address this issue, the back-projection has been investigated [13]. Specifically, a deep back-projection network (DBPN) is developed to improve the learning performance. However, in DBPN, the dependencies between LR and HR images are still not fully exploited, since it does not consider the loss between the down-sampled image and the original LR image. As a result, the representation capacity of deep models may not be well exploited.
We seek to address the above issues in two directions. First, we devise a novel gradient sensitive loss relying on the gradient magnitude of an image. To do so, we hope to well recover both lowand high-frequency information at the same time. To achieve high performance of SR, besides the pixel-wise PSNR, we also seek to achieve high PSNR score over image gradients. Second, by exploiting the duality in LR and SR images, we formulate the SR problem as a dual learning task and we present a dual reconstruction network (DRN) by introducing an additional dual module to exploit the bi-directional information of LR and HR images.
In this paper, we make the following contributions. First, we devise a novel gradient sensitive loss to improve the reconstruction performance. Second, we develop a dual learning scheme for image super-resolution by exploiting the mutual dependencies between low-and high-resolution images via the task duality. Third, we theoretically prove the effectiveness of the proposed dual reconstruction scheme for SR in terms of generalization ability. Our result on generalization bound of dual learning is more general than [46]. Last, extensive experiments demonstrate the effectiveness of the proposed gradient sensitive loss and dual reconstruction scheme.

Related work
Super-resolution. One classic SR method is the interpolation-based approach, such as cubicbased [16], edge-directed [1,26] and wavelet-based [36] methods. These methods, however, may oversimplify the SR problem and usually generate blurry images with overly smooth textures [25,42]. Besides, there are some other methods, such as sparsity-based techniques [12,19] and neighborhood embedding [11,41], which have been widely used in real-world applications.
Another classic method is the reconstruction-based method [3,5,29], which takes LR images to reconstruct the corresponding HR images. Following such method, many CNN-based methods [18,21,30,40,42,43,50,51] were developed to learn a reconstruction mapping and achieve state-ofthe-art performance. However, all these methods only consider the information from HR images and ignore the mutual dependencies between LR and HR images. Very recently, Haris et al. [13] propose a back-projection network and find that mutual dependencies are able to enhance the performance of SR algorithms.
Loss function. The loss function plays a very important role in image super-resolution. The mean squared error (MSE) [8,17,21] and mean absolute error (MAE) [51] are two widely used loss functions: where · 1 denotes 1 -norm, and I and I denote the ground-truth and the predicted images. While MSE is a standard choice which is directly related to PSNR [8], MAE may be a better choice to produce sharp results [28]. Nevertheless, the two loss functions can be used simultaneously. For example, in [18] they first train the network with MAE and then fine-tune it by MSE . Recently, Lai et al. [24] and Liao et al. [27] introduce the Charbonnier penalty which is a variant of MAE . Justin et al. [20] propose a perceptual loss, by minimizing the reconstruction error based on the extracted features, to improve the perceptual quality. More recently, Ledig et al. [25] leverages the adversarial loss to produce photo-realistic images. However, these methods 2 take the whole image as input and do not distinguish between low-and high-frequency details. As a result, the low-frequency content and high-frequency structure information cannot be fully exploited.

Gradient sensitive loss
In this section, we propose a gradient-sensitive loss in order to preserve both low-frequency content and high-frequency structure of images for image super-resolution. As aforementioned, optimizing the pixel-wise loss often lacks high-frequency structure information and may produces perceptually blurry images. To address this issue, we hope to recover the image gradients as well in order to capture the high frequency structure information. Intuitively, one may exploit the loss over gradients [31]: where ∇ x I and ∇ y I denote the directional gradients of I along the horizontal (denoted by x) and vertical (denoted by y) directions, respectively. Apparently, we cannot directly minimize G for SR. Instead, we can construct a joint loss by considering both pixel-level and gradient-level errors: where P is the pixel-level loss, which can be either MSE or MAE and λ is a parameter to balance the two terms. Minimizing GP in (3) will help to recover the gradients, but it means we have to sacrifice the accuracy over the pixels, and a good balance is often hard to made. In other words, the reconstruction performance in terms of PSNR over the image shall degrade when considering the recovery of gradients.
A natural questions arises: given an image, can we find a way to separate the high-frequency part from its low-frequency part and then impose losses over the two parts separately? If the answer is positive, the emphasis on the gradient-level loss will not affect the pixel-level loss and then the dilemma within GP shall be addressed. Here, we develop a simple method concerning the above question. Specifically, we seek to find a mask M to decompose the image I by Relying on the directional gradients ∇ x I H and ∇ y I H , we can easily devise such a mask. In fact, given the gradient magnitude G, where G i,j = (∇ x I i,j ) 2 + (∇ y I i,j ) 2 , we can define the mask as the normalization of G into [0, 1]: where min(G) and max(G) denote the minimum and maximum value in G, respectively. It is clear that M I and (1 − M) I represent the low and high-frequency parts, separately. Finally, we define our gradient-sensitive loss as where denotes the element-wise multiplication and λ is a trade-off parameter. Here, we adopt MAE as the pixel-level loss P .
In Eqn. (5), the gradient-level loss G (M I, M I) focuses on the high-frequency part and it helps to improve the reconstruction accuracy of gradients. Different from GP in (3), in GS , the pixel-level loss P focuses on low-frequency part, since the gradient information has been subtracted from I. As a result, the reconstruction accuracy over pixels will not suffer even though we put emphasis on gradient. Most importantly, since the gradient information can be well recovered, it will help to improve the overall performance in terms of PSNR and visual quality significantly. 3

Dual reconstruction network
Most existing methods employ feed-forward architecture and focus on minimizing the reconstruction error between recovered image and the ground-truth. As such, they ignore the mutual dependencies between LR and HR images. As a result, the representation capacity are not fully exploited [13,47].
Here, we seek to investigate the duality in SR problems and propose a dual reconstruction scheme to fully exploiting the mutual dependencies between LR and HR images to improve the performance.  Figure 1: Demonstration of the network architectures for 4× super-resolution. The blue lines denote the standard reconstruction for the primal model and the red lines denote the backward shortcut for the dual model.

Dual reconstruction scheme for super-resolution
Dual supervised learning (DSL) has been investigated in [14,47] and shows that DSL can improve the practical performances of both tasks. Inspired by [47], we introduce an additional dual reconstruction of LR to improve the primal reconstruction of HR images. We aim to simultaneously learn the primal mapping P (·) to reconstruct HR images and the dual mapping D(·) to reconstruct LR images. Let x ∈ X be LR images and y ∈ Y be HR images. We formulate the SR problem as a dual reconstruction learning scheme as below.
Definition 1 (Primal learning task) The primal learning task aims to find a function P : X → Y, such that the prediction P (x) is similar to its corresponding HR image y.
Definition 2 (Dual learning task) The dual learning task aims to find a function P : Y → X , such that the prediction of D(y) is similar to the original input LR image x.
If P (x) were the correct HR image, then the down-sampled images D(P (x)) should be very close to the input LR images x. In other words, the dual reconstruction of LR images is able to provide additional supervision to learn a better primal reconstruction mapping. To train the proposed model, we construct a dual reconstruction loss which can be computed as follows: where 1 (·) and 2 (·) denote the loss function for primal and dual reconstruction tasks, respectively.

Progressive dual reconstruction for super-resolution
We build our network based on the proposed dual reconstruction scheme to exploit the mutual dependencies between LR and HR images, as shown in Figure 1. Following the design of progressive reconstructions [24,44], the proposed model consists of multiple dual reconstruction blocks and progressively predicts the images from low-resolution to high-resolution. Let r be the upscaling factor, the number of the blocks depends on the upscaling factor: L = log 2 (r). For example, the model contains 2 blocks for 4× and 3 blocks for 8× upscaling.
The dual reconstruction scheme can be easily implemented by introducing a backward shortcut connection (see red lines in Figure 1). In each block, the primal model P consists of multiple residual modules [15] followed by a sub-pixel convolution layer to increase the resolution by 2× upscaling. Since the dual task aims to learn a much simpler downsampling operation compared to the primal upscaling mapping, the dual model D only contains two convolution layers and a ReLU [34] activation layer. During training, we use the bicubic downsampling to resize the ground truth HR image y to y l in l-th block. Let y l be the predicted image of l-th block and y 0 = y 0 be the input LR image at the lowest level. For convenience, let {θ P } and {θ D } be the parameters for primal and dual models at all levels. We build a joint loss to receive the supervision at different scales: where P l and D l denote the primal and dual model in l-th block, respectively. We set both 1 and 2 on the primal and dual reconstruction tasks to the proposed GS loss function.

Theoretical analysis
We theoretically analyze the generalization bound for the proposed method, where all definitions, proofs and lemmas are put in Appendix A, due to the page limitation. The generalization error of the dual learning scheme is to measure how accurately the algorithm predicts for the unseen test data in the primal and dual tasks. In particular, we obtain a generalization bound of the proposed model using Rademacher complexity [4].
, and the hypothesis set H dual be infinite. Then, for any δ > 0, with probability at least 1 − δ, the generalization error E(P, D) (i.e., expected loss) satisfies for all (P, D) ∈ H dual : where m is the sample number and E(P, D) is the empirical loss, while R DL m and R DL Z represent the Rademacher complexity and empirical Rademacher complexity of dual learning, respectively.
This theorem suggests that using the hypothesis set with larger capacity and more samples can guarantee better generalization. We highlight that the derived generalization bound of dual learning, where the loss function is bounded by [0, M ], is more general than [46].
Remark 1 Based on the definition of Rademacher complexity, the capacity of the hypothesis set H dual ∈P×D is smaller than the capacity of hypothesis set H∈P or H∈D in traditional supervised learning, i.e., R DL Z ≤ R SL Z , where R SL Z is Rademacher complexity defined in supervised learning. In other words, dual reconstruction scheme has a smaller generalization bound than the primal feed-forward scheme and the proposed dual reconstruction model helps the primal model to achieve more accurate SR predictions. 4

Experiments
In the experiments, we perform super-resolution to recover images that are downsampled by factors of 4 and 8, respectively. We compare the performance of the proposed method with several stateof-the-art methods on five benchmark datasets, including SET5 [6], SET14 [49], BSDS100 [2], URBAN100 [17] and MANGA109 [32]. For quantitative evaluation, we adopt two common image quality metrics, i.e., PSNR and SSIM [45] in the paper.

Implementation details
We train the proposed DRN model using a random subset of 350k images from the ImageNet dataset [37]. We randomly crop the input images to 128 × 128 RGB images as the HR data, and downsample the HR data using bicubic kernel to obtain the LR data. We use ReLU activation in both the primal and dual reconstruction model. Each reconstruction block in the primal model has 7 identical residual modules, i.e., 14 modules for 4× and 21 modules for 8× upscaling. We adopt the sub-pixel convolutional layer [38] to increase the resolution by 2× upscaling. The hyperparameter λ in Eqn. (5) is set to 2. During training, we apply the Adam algorithm [23] with β 1 = 0.9. We set minibatch size as 16. The learning rate is initialized to 10 −5 and decreased by a factor of 10 for every 5 × 10 5 for total 10 6 iterations. All experiments were conducted using PyTorch.

Demonstration of gradient sensitive loss
In this part, we perform super-resolution with 4× upscaling to study the impacts of different losses, including the Mean Square Error (MSE), Mean Absolute Error (MAE), image gradient loss [31], and the proposed gradient-sensitive (GS) loss. Figure 2 presents    the results over the RGB images. The proposed gradient-sensitive loss converges to the highest PSNR score among all the compared losses. In addition, the gradient magnitude map obtained by GS is more close to the ground-truth compared with the other losses. From the reconstructed RGB images, we observe that GS is able to capture more details and maintain the perceptual fidelity of the original HR images.

Comparisons with state-of-the-art methods
We compare the performance of our proposed DRN approach with several state-of-the-art methods, including Bicubic, SRCNN [9], SelfExSR [17], DRCN [22], ESPCN [38], SRResNet [25], SR-GAN [25], FSRCNN [10], VDSR [21], DRRN [40], LapSRN [24], SRDenseNet [42] and EDSR [28]. While preparing this paper, we are aware of a very recent work [13] which shows promising performance. For fair comparison, we train the model using their souce code on our data set with the same setting. However, our reproduced results are worse than the reporting results in [13]. One possible reason is that they use more training data. We study the effect of the number of training data in Figure 5 of supplementary file.
Tables 1 and 2 present the results of 4× and 8× image super-resolution, respectively. For the 4× super-resolution tasks, our proposed GS loss and DRN approach outperform the other conducted methods on most datasets. For the 8× super-resolution tasks, DRN and GS loss achieve the best and the second best performance among all the conducted methods, respectively. These observations demonstrate the effectivenes of the proposed methods. In addition, DRN with GS loss outperforms GS loss on all the datasets, which validates that the proposed dual reconstruction mechanism is able to further improve the performance. For further comparison, we provide visual comparisons on some reconstructed images. Figures 3 shows the 4× and 8× SR images obtained by different methods and the corresponding metrics, respectively. We observe that our proposed DRN method consistently achieves the best numerical results and the best visual quality.   Table 3 lists the PSNR scores over the image gradient on several benchmark datasets. Our proposed DRN achieves the best performance, which demonstrates that the DRN network has a better ability to capture the structural information compared with the other methods.
6.2 Effects of dual reconstruction scheme and progressive structure.
In this experiment, we evaluate the effects of the dual reconstruction scheme and conduct analysis on the progressive structure. The "non-progressive" methods directly predict the final HR images without the supervision from the prediction of intermediate images. The "non-dual" learning methods remove the dual learning part and fall back to plain feed-forward methods. Table 4 shows the PSNR  scores of the 4× super-resolution tasks on the SET5 and SET14 datasets. We observe that both the progressive and dual methods outperform the plain methods ("non-progressive" and "non-dual") The combination of dual reconstruction scheme and progressive structure method achieves the best performance. These results demonstrate the efficacy of the proposed progressive reconstruction and dual learning approaches.

Conclusion
In this work, we propose a novel gradient sensitive loss (GS) to capture both low-frequency content and high-frequency structure for image super-resolution. Moreover, to exploit the mutual dependencies between LR and HR images, we propose a dual reconstruction to further improve the performance. Our model is trained with the proposed GS loss in a progressive coarse-to-fine manner. More critically, we conduct theoritical analysis on the generalization bound of the proposed method. Extensive experiments demonstrate that the proposed method produces perceptually sharper images and significantly outperforms the state-of-the-art SR methods with a large upscaling factor of 4× and 8×.

Supplementary Materials for "Dual Reconstruction Nets for Image Super-Resolution with Gradient Sensitive Loss" A Theoretical analysis
In this section, we will analyze the generalization bound for the proposed method. The generalization error of the dual learning scheme is to measure how accurately the algorithm predicts for the unseen test data in the primal and dual tasks. Firstly, we will introduce the definition of the generalization error as follows: Definition 3 Given an underlying distribution S and hypotheses P ∈ P and D ∈ D for the primal and dual tasks, where P = {P θxy (x); θ xy ∈ Θ xy } and D = {D θyx (y); θ yx ∈ Θ yx }, and Θ xy and Θ yx are parameter spaces, respectively, the generalization error (expected loss) of h is defined by: In practice, the goal of the dual learning is to optimize the bi-directional tasks. For any P ∈ P and D ∈ D, we define the empirical loss on the m samples as follows: Following [4], we define Rademacher complexity for dual learning in this paper. We define the hypothesis set as H dual ∈ P × D, this Rademacher complexity can measure the complexity of the hypothesis set, that is it can capture the richness of a family of the primal and the dual models. For our application, we mildly rewrite the definition of Rademacher complexity in [33] as follows: Definition 4 (Rademacher complexity of dual learning) Given an underlying distribution S, and its empirical distribution Z = {z 1 , z 2 , · · · , z m }, where z i = (x i , y i ), then the Rademacher complexity of dual learning is defined as: where R Z (P, D) is its empirical Rademacher complexity defined as: where σ = {σ 1 , σ 2 , · · · , σ m } are independent uniform {±1}-valued random variables with p(σ i = 1) = p(σ i = −1) = 1 2 .

A.1 Generalization bound
This subsection give a generalization guarantees for the dual learning problem. We start with a simple case of a finite hypothesis set. This theorem shows that a larger sample size m and smaller hypothesis set can guarantee the generalization. Next we will give a generalization bound of a general case of infinite hypothesis sets using Rademacher complexity.
Theorem 3 Let 1 (P (x), y) + 2 (D(P (x)), x) be a mapping from X × Y to [0, M ], then for any δ > 0, with probability at least 1 − δ, the following inequality holds for all (P, D) ∈ H dual : Proof 2 Based on Theorem 3.1 in [33], we extend a case for 1 (P (x), y) Theorem 3 shows that with probability at least 1 − δ, the generalization error is smaller than . It suggests that using the hypothesis set with larger capacity and more samples can guarantee better generalization. Moreover, the generalization bound of dual learning is more general for the case that the loss function 1 (P (x), y) + 2 (D(P (x)), x) is bounded by [0, M ], which is different from [46].
Remark 2 Based on the definition of Rademacher complexity, the capacity of the hypothesis set H dual ∈P×D is smaller than the capacity of hypothesis set H∈P or H∈D in traditional supervised learning, i.e., R DL Z ≤ R SL Z , where R SL Z is Rademacher complexity defined in supervised learning. In other words, dual learning has a smaller generalization bound than supervised learning and the proposed dual reconstruction model helps the primal model to achieve more accurate SR predictions.

B.1 Demonstration of gradient sensitive loss
For better understanding, we plot some results about our manipulation of image gradient in Figure 4. By visualizing the gradient ∇I of the image I, we can observe the structure information directly. Meanwhile, M I means the mask M has the pixel-wise multiplication with image I and (1−M) I represent the rest part. In our proposed method, we use the gradient of M I, i.e. ∇(M I), to compute the loss function. Table 5 summarizes the characteristics of different loss functions and training schemes. In particular, the standard training scheme only forces the model to match the HR images, while the dual scheme receives the supervised information from both LR and HR images.

B.2 Comparisons of different loss functions and training schemes
For the different objective functions, the perceptual loss does not obtain frequency information from images, the gradient loss only captures the high-frequency information, and the MAE, MSE and adversarial loss only btains the low-frequency information. In comparison, our proposed gradientsensitive loss ( GS ) is able to capture both the low-and high-frequency information from images.
Overall, the proposed DRN method with the dual scheme is able to exploit both the low-and high-frequency information, and receive supervision form both LR and HR images.

M⨀I
(1 − M)⨀I I ∇ I Figure 4: Demonstration of the mask M. In this experiment, we study the performance of our proposed DRN method under different values of the parameter λ. From Table 6, when the parameter λ is too small, the method cannot achieve promising performance, since the gradient-level loss only captures the structural information. When the parameter λ increases monotonically, the performance of DRN increases gradually. This demonstrates that the combination of the gradient-level loss and the pixel-level loss is effective to achieve promising results. In our setting, we empirically set λ = 2, since we find that a larger value usually does not bring further performance improvement.

C.2 Effect of training data ratio
We conduct an experiment on SET5 to evaluate the influence of the number of training data. From Figure 5, when increasing the ratio of training data on the whole dataset, the values of PSNR score increases gradually. In addition, DRN consistently outperforms DBPN on all the data ratios.

C.3 Comparison of model complexity
We report the PSNR scores and the numbers of the parameters in DRN and several state-of-the-art models on 4× and 8× SR in Figures 6 and 7, respectively. The x-axis represents the number of the model parameters, and the y-axis means the value of PSNR. The results show that the proposed DRN method can achieve the best performance on both two datasets with the lowest computational complexity compared with the other baseline methods.

C.4 More results
For further comparison, we provide the experimental results compared with the baseline methods for 8× SR on several benchmark datasets.  Figure 8: More results of visual comparison for 8× upscaling super-resolution.