{'fraction_non_alphanumeric': 0.034716591729932925, 'fraction_numerical': 0.0116090513746591, 'mean_word_length': 4.796838282418286, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform

Single image super-resolution(SISR) is an ill-posed problem that aims to obtain high-resolution (HR) output from low-resolution (LR) input, during which extra high-frequency information is supposed to be added to improve the perceptual quality. Existing SISR works mainly operate in the spatial domain by minimizing the mean squared reconstruction error. Despite the high peak signal-to-noise ratios(PSNR) results, it is difficult to determine whether the model correctly adds desired high-frequency details. Some residual-based structures are proposed to guide the model to focus on high-frequency features implicitly. However, how to verify the fidelity of those artificial details remains a problem since the interpretation from spatial-domain metrics is limited. In this paper, we propose FreqNet, an intuitive pipeline from the frequency domain perspective, to solve this problem. Inspired by existing frequency-domain works, we convert images into discrete cosine transform (DCT) blocks, then reform them to obtain the DCT feature maps, which serve as the input and target of our model. A specialized pipeline is designed, and we further propose a frequency loss function to fit the nature of our frequency-domain task. Our SISR method in the frequency domain can learn the high-frequency information explicitly, provide fidelity and good perceptual quality for the SR images. We further observe that our model can be merged with other spatial super-resolution models to enhance the quality of their original SR output.


Introduction
Single image super-resolution(SISR) aims to recover high-frequency details for a high-resolution(HR) image from one of its degraded low-resolution(LR) version. After years of development, the SISR has been widely used in many computer vision tasks, such as media content enhancement [1], medical imaging [2] and satellite imaging [3]. Traditional state-of-the-art SR methods mainly adopt the example-based [1] strategy, exploiting internal similarities or learning a mapping from the external dictionary. The sparse-coding-based SR [4] is one of the most representative methods.
Recently, deep convolutional neural network (CNN) based SISR methods have achieved significant improvements over traditional methods. Deep learning-based methods treat this problem as a dense image regression task, which learns an end-to-end image mapping function represented by a CNN between LR and HR images. Dong et al. [5] proposed SRCNN that first adopted deep learning into SISR using a three-layer CNN to represent the mapping function. Residual block [6] was later introduced into SISR in SRResNet [7] and improved in EDSR [8]. Residual block makes it possible to build deeper or wider networks. Zhang et al. [9] and Tong et al. [10] adopted dense blocks [11] to combine features from different levels. Zhang et al. [12] improved residual block by adding channel attention. Based on the progress of non-blind methods, blind super-resolution methods [13], which aim at complex degradation models in real scenarios, have received increasing attention recently.
The SISR methods mentioned above commonly use the minimization of the mean squared error (MSE) between the recovered SR image and the HR ground truth as the optimization target. Minimizing spatial MSE also maximizes the peak signal-to-noise ratio (PSNR), which is a common measure used to evaluate SR algorithms. However, such a pipeline often results in blurry effects because the high-frequency textures have been excessively destructed in the degrading process and are hard to predict. Generative adversarial networks (GANs) [14] based SISR approaches are proposed to relieve the above problems. However, the unpleasant hallucinations and artifacts caused by GANs further pose more challenges. Zhang et al. [12] further proposed a residual-in-residual (RIR) structure to bypass the redundant lowfrequency information through multiple skip connections, implicitly guiding the network to focus on learning high-frequency information. However, since the commonly used PSNR and structural similarity index measure(SSIM) are based on per-pixel loss and picture global information, respectively, their perception of high-frequency details is limited. To the best of our knowledge, current spatial domain-based methods do not have an explicit approach for learning high-frequency information and verifying the fidelity of output artificial details.
To practically resolve this problem, we propose FreqNet, a frequencydomain-based super-resolution network, to directly learn the reconstruction of high-frequency features. The proposed network contains two parallel flows: the Spatial Extraction Network(SEN) and the Frequency Reconstruction Network(FRN), in order to make use of both domains' information. We first convert both LR and HR images to frequency coefficients using discrete cosine transform (DCT) [15], then reshape them to obtain DCT feature maps. The SEN takes standard LR images as input, through the spatial feature reconstruction trunk and the down-sampling shrinking trunk to obtain one component of target HR DCT feature maps. The FRN is purely operated on frequency domain, which takes LR DCT feature maps as input, through the frequency-domain reconstruction trunk to obtain the other component. The weighted sum of two components makes our final frequency domain output, which can be converted to SR image through inverse discrete cosine transform(iDCT) [15]. Thanks to the characteristic of DCT, we can easily merge our output with any other SR model to enhance the high-frequency details of its output. We further propose depth-wise residual block(DWRB) and deformable residual block(DRB) to be implemented respectively in FRN and SEN that can better use the characteristics of the frequency domain feature maps. As the ability of spatial MSE (and PSNR) to capture highfrequency detail is very limited, we propose a frequency-domain loss function to evaluate the quality of the output SR image.
Overall, our contributions are three-fold: (1) We propose FreqNet, a frequency-domain-based SISR network, to learn the high-frequency features explicitly with a specially designed pipeline. Our network can produce perceptually satisfying results with high-fidelity details. (2) We propose depthwise residual block structure and deformable residual block structure to fit the nature of frequency-domain feature extraction. Both structures can improve our network's reconstruction quality and feature extraction ability. (3) We propose a frequency-domain loss function and a corresponding metric that measures output quality from the accuracy of high-frequency detail reconstruction.

Related Works
Numerous image deep learning-based SR methods have been studied in the computer vision community. Here we focus on works related to CNNbased methods and the works on frequency-domain learning.

Image Super-Resolution with CNN
Numerous methods have proven the effectiveness of the CNN-based pipeline on image super-resolution tasks. The pioneering work was done by Dong et al. [5], their proposed SRCNN for image SR achieved superior performance against previous works. Kim et al. proposed VDSR [16] and DRCN [17] by introducing residual learning to ease the training difficulty and significantly improve accuracy. Tai et al. introduced recursive blocks in DRRN [18] and memory blocks in MemNet [19]. A faster network structure FSRCNN [20] was proposed to accelerate the pipeline of SRCNN. Ledig et al. [7] introduced ResNet[6] to construct a deeper network, SRResNet, for image SR. They also proposed SRGAN with perceptual losses [21] and generative adversarial network (GAN) [14] for photo-realistic SR. Such GAN based model was then introduced in ESRGAN [22], which confirmed that dropping the batch normalization layers can result in better performance. Although SRGAN and ESRGAN can alleviate the blurring and over smoothing artifacts, their predicted results may not be faithfully reconstructed and produce unpleasing artifacts. By removing unnecessary modules in conventional residual networks, Lim et al. [8] proposed EDSR and MDSR, which achieve significant improvement. Zhang et al. [12] introduce channel attention to residual block. Blind-SR methods have also received increasing attention recently, aiming at complex degradation models in real scenarios by estimating degradation kernel using an extra module [23]. The methods of [24], [25] and [26] achieved state-of-the-art performance in real-world scenario with multiple modelling strategies.
However, all these CNN-based methods operate on the spatial domain. The information on the frequency domain is not directly used, though the recovery of high-frequency information is precisely the target of the image super-resolution task.

Frequency-Domain based Deep Learning
Projecting image to frequency domain provides a new perspective for various computer vision tasks. Remarkable performance has been achieved in some frequency-domain works. Works of [27] , [28] and [29] jointly train autoencoder-based networks on compression and inference tasks with frequencydomain input. [30] extracts features from the frequency domain to classify images. [31] proposes a model conversion algorithm to convert the spatialdomain CNN models to the frequency domain. [32] propose a method of learning in the frequency domain using DCT-based sparse image representations, proving that we can use frequency-domain information directly in current CNN models without a complex model transition procedure. [33] further translate the DCT representation into a sequence of DCT channel, spatial location, and DCT coefficient triples, and achieve state-of-art performance on image generation and restoration tasks with a Transformer-based auto-regressive architecture.
The essence of the SR task is to recover the information of high-frequency channels in the image. Hence, frequency-domain features are informative for HR reconstruction and can potentially enhance the performance with proper methods. However, there is no existing SR method using the characteristics of DCT feature maps. Hence, we propose a super-resolution pipeline on the DCT domain, which we will present in detail in the following section.
Precisely, the overall architecture is given in figure 1. In section 3.1, we introduce the image conversion process that projects the spatial image to spatial domain. In section 3.2 and 3.3, we explain the architecture and components of FreqNet in detail. The propsed frequency-domain loss function will be presented in section 3.4. We propose a frequency-domain based pipeline for 4× image super-resolution. Our method consists of an image conversion process that converts the spatial image to the frequency domain and a specialized network for training with the frequency domain information. As shown in 1, our proposed network consists of two parallel sub-network, respectively operates on spatial-domain and frequency-domain inputs to make use of both domains' information. We will first explain the image conversion process in the following section. Details of architecture will be discussed in 3.2.

Image Conversion to the Frequency Domain
Following the JPEG codec, we first transform the original RGB images to zero-centered normalized YCrCb color space, containing a brightness component Y (luma) and two color components Cb and Cr (chroma). Then we upsample the LR image to make it the same size as the HR image. To get frequency-domain information, we crop the images into uniform size of pixel blocks, then pass them through Discrete Cosine Transform(DCT) module. The DCT projects an image into a collection of cosine components which stands for different frequencies of 2D signals. Given a block size M , the two-dimensional DCT converts a zero-centered M × M pixel blocks P to obtain an M × M DCT block D, as interpreted below:

Generation of DCT Blocks
Where u and v are the horizontal and vertical index of frequencies in the DCT block, i and j stand for the horizontal and vertical index of pixel block, and α is a normalizing scale factor to enforce orthonormality. For a standard DCT transform in JPEG codec, the block size M is 8, which indicates that any information in an 8 × 8 pixel block can be represented by a linear combination of 64 2D signals. However, in 4× image super-resolution, the 8 × 8 block is upscaled to 32 × 32. Thus we perform DCT transform with the block size 32. At this stage, both the LR and HR images are converted into frequency-domain blocks that contain the DCT information of 1024 frequency channels.

Reforming DCT feature maps
Not all information in the 32 × 32 frequency range can be perceived for the perceptual ability of the human eye. Many other DCT based meth- ods induce sparsity to DCT blocks through quantization. However, for the super-resolution task, we tend to preserve the information as much as possible. Thus, as illustrated in figure 3, we perform a region selection on the DCT block. Only the values inside the left-top R×R selected region will be preserved for the next step. In practice, we choose R = 10 as a trade-off between training difficulty and perceptual quality. We will explain later how we handle the values outside the selected region.
Following the processing method proposed by [32], we flatten the DCT blocks to DCT vectors of length 1 × R 2 . Then we pose these vectors at their corresponding spatial positions, forming a cuboid of size H/R × W/R × R 2 , where H and W are the height and width of the original image. This cuboid is a collection of DCT feature maps, each channel at the third dimension is a frequency-domain feature map that contains the information of the frequency it represents.

Channel-wise Normalization
We further perform normalization on each frequency channel. For channel i of the frequency-domain feature maps M , we perform: Where M ean i and Std i denotes the mean and standard deviation of channel i that are pre-calculated on our training set.
Unlike quantization, this normalization process does not change the relative intensity of each feature map, thus guaranteeing the integrity of information. The purpose of this operation is to project the values to a suitable range for learning.

Architecture of FreqNet
As shown in figure 1, our FreqNet contains two parallel data flows: the Spatial Extraction Network(SEN) and the Frequency Reconstruction Network(FRN), in order to make use of both domains' information.
The SEN takes up-scaled LR image I LR as input. Only one convolutional layer is used to extract the shallow feature F shallow from the LR input. F shallow is then passed through the Reconstruction Trunk(RT), which contains a sequence of multiple Residual Groups(RG) [12] and Deformable Residual Groups(DRG) to convert the spatial feature maps into frequency domain features F f req . Then we feed F f req to Shrinking Trunk(ST), which consists of 4 down-sampling convolution layers with stride = 2, to gradually shrink the scale of features maps while maintaining the channels. The final output M SR 1 is one component of target HR DCT feature maps. The overall process can be interpreted as: Where H shallow denotes the first convolution operation, H RT and H ST denote the RT and ST structure. The FRN is purely operated on frequency domain. We take the preprocessed LR DCT feature maps M LR as input, through the frequencydomain reconstruction trunk(FRT), which contains a sequence of depth-wise residual groups(DWRG) and RG to obtain the other component of target HR DCT feature maps, noted as M SR 2 . A skip-connection is added to take advantage of the similarities between input and the target, thus drawing attention towards the difference on high-frequency channels. The overall process can be interpreted as: Where H F RT denotes the FRT structure.
The outputs of two sub-network have the same size, and a weighted element-wise sum is applied to get the final output: Where the W 1 and W 2 are the pre-defined weights for two components.
The output M SR is further fed to a 2-stage inverse Discrete Cosine Transform(iDCT) module, which is an inverse flow of data-processing pipeline we defined in 3.1. We first project the M SR back to its original range of values by performing denormalization on each channel. Then, in stage-1, we reform the H/R × W/R × R 2 DCT feature maps back to DCT blocks of size R × R, and the rest of 32 × 32 block is filled with information from LR DCT blocks. Then we use iDCT to get the final SR image in stage-2. Inspired by the success of residual groups(RG) in [12], we take it as the basic module of our network. As shown in figure 4, an RG is a sequence of n residual blocks(RB) [8] with an in-group skip connection between the input and output features. The original RB can be interpreted as:

Modified Residual Group
Where H Conv denotes a convolution layer, F input is the feature from last block and F output is the feature towards next layer. As described in 3.1.3, the final output of network should be of zero-centered distribution, thus we replace ReLU layer by LeakyReLU with a high negative slope to fit our case.

Deformable Residual Group
The RG structure makes it possible to achieve large depth, consequently providing a large receptive field size. However, uniformly extending the receptive field does not always positively impact high-precision required tasks, such as the reconstruction of frequency-domain feature maps, due to the potential redundant information. Deformable convolution layer [34](DefConv) can be a solution. By learning an offset, DefConv provides the ability to constrain the sampling area. Each convolution operation only focuses on the valuable region, reducing the impact from the redundant receptive area.
Thus, as shown in figure 4 we further integrate DefConv into RB by partly replacing the original convolutional layers, introducing the deformable residual block(DRB), which is the basic module of deformable residual group(DRG): Where H Def Conv denotes the deformable convolution layer. The proposed DRB structure has better guidance on the receptive field, thus yield more accurate feature extraction from last layer. We implement DRG sequence in the reconstruction trunk of SEN sub-network, after a sequence of regular RG, to improve the robustness of reconstructed F f req .

Depth-wise Residual Group
For most spatial domain tasks, the intermediate deep feature maps are abstract and strongly correlated. However, through the reforming method we defined in 3.1.2, the frequency-domain feature maps have concrete semantic information and share less correlation between each other. To better reflect this characteristic, we propose the depth-wise residual block(DWRB) that replace the first convolution layer in RB by depth-wise convolution layer [35]: Where H DW Conv denotes the depth-wise convolution layer. A depth-wise convolution layer performs 2-D convolution on each channel of the input without merging information from other channels, which is suitable to make the module focus on extracting information from own channel for the next stage of reconstruction, rather than relying on global information. Depthwise residual group(DWRG) is the RG that deploy the DWRB instead of RB.

Frequency-domain Loss Function
The definition of our frequency-domain loss function L f req is critical to the performance of our network. Commonly, the loss function of superresolution task is based on pixel-wise Mean Square Error(MSE), as minimizing spatial MSE also maximizes the peak signal-to-noise ratio(PSNR). However, solutions from MSE optimization can achieve high PSNR while lacking high-frequency content, which results in unsatisfying perceptual quality with overly smooth textures [7].
For our frequency domain super-resolution, this problem can be solved in a intuitive method. Since the target is a series of frequency-domain feature maps with semantic meaning assigned to each channel, we can allocate different weights to each frequency channel while computing the loss, thus explicitly guide the network to focus on the reconstruction of selected highfrequency channels. Following [36], we further replace the MSE backbone by Charbonnier Loss that can better handle the outliers, which are more likely to appear in frequency-domain samples. The proposed frequency-domain loss function L f req is calculated as: Where L Char is the backbone of Charbonnier Loss, W map and H map denotes the width and height of output feature maps, β c denotes the weight assigned to channel c and M denotes the frequency-domain feature maps, as we previously define in equation 8.

Experimental Settings
Our experimental settings about datasets, degradation models, evaluation metric and training settings are declared below: Datasets and degradation model. Following [37] We use 800 training images from DIV2K dataset [37] as training set. For testing, we use four standard benchmark datasets: Set5 [38], Set14 [39], BSD100 [40] and MANGA109 [41]. We conduct experiments with Bicubic degradation model.
Evaluation Metrics. The SR results are evaluated with PSNR on the luminance channel(Y channel) of transformed YCrCb space. We also propose a frequency-domain reconstruction metric(FRM) on the luminance channel that measures the quality of high-frequency feature reconstructed: Training Settings. We crop 800 training images into mini patches. Respectively, the size of cropped LR image is 32 × 32 and the size of cropped HR image is 256 × 256. The relative location of each pair of LR and HR patches is strictly identical. Our model is trained by ADAM optimizor [42], with β 1 = 0.9, β 2 = 0.99 and = 10 − 8. We implement Cosine Learning Rate(CosLR) strategy, which periodically adjust the learning rate at t th epoch of i th period with the equation: Where the η max is 10 − 4 and η min is 10 − 7, the number of epochs in each period is 30. We use PyTorch [43] to implement our method with Nvidia Geforce RTX 2080 ti GPU. The channel-wise weights allocation of our proposed loss function L f req (Equation 13) will be discussed in detail in section4.3.

Results with Bicubic Degradation Model
We quantitatively compare our method with 8 State-of-the-art methods, including SRCNN [5], FSRCNN [20], EDSR [8], EDN [9], RRDB [22] and its perceptual-driven method ESRGAN [22], MSRResNet [7] and its perceptualdriven method MSRResNet-GAN [7]. We further perform visual comparisons with these two GAN-based methods and their PSRN-oriented version to demonstrate the perceptual quality and fidelity of the output from our model. Table 1 shows quantitative comparisons for our 4× SR task, we compare the average PSNR and FRM on Y channel. The PSNR results of ESR-GAN and MSRResNet pair are computed using the released model. For the other models, the results are cited from their papers. All the FRM results are computed using released models. Our model has the best FRM with a slight decrease in PSNR value, which shows that our method has a more accurate reconstruction of key high-frequency information. Meanwhile, although GAN-based methods visually provide more high-frequency details, their FRM values are generally low, reflecting the lack of accuracy of highfrequency information reconstructed by such methods. We will discuss the visual behavior in detail in section 4.2.2.

Effects of Frequency-domain Loss Function and Modified RG
We study the effects of proposed Deformable Residual Group, Depth-wise Residual Group and the Frequency-domain Loss Function.

Settings and Effects of Frequency-domain Loss Function.
As we defined in Equation 13, each channel has a pre-assigned weight. We propose a statistical solution to decide the weight of each channel coarsely. As shown in Figure 6, given a pair of HR and up-sampled LR DCT blocks, after the region selection process(i.e. Figure 3, (d)), of size R × R, we perform 8 times of computation in total. For the i-th computation, we keep the (i + 2) × (i + 2) DCT region at the left-top of original DCT block unchanged, and set the values outside the selected region to be 0. Then we perform iDCT on both LR and HR DCT blocks and compute the mean pixel-wise residual res i of two converted pixel blocks as: Region: 3 4-3 5-4 6-5 7-6 8-7 9-8 10-9 β c 1 1 5 10 10 5 1 1 We randomly picked 1000 samples from the training set to perform the statistics by accumulating res i . We define res 0 = 0, then for each i, the value v i = res i − res i−1 reflects the difference between HR and LR images while considering the addition frequency channels of R = i + 2, which is proportional to their importance. Therefore, based on [v i ] i∈ [1,8] , we allocate weights as the table 2 shows, where Region i − (i − 1) denotes the additional channels between the left-top i × i region and (i − 1) × (i − 1) region of the R × R DCT block, and β denotes the weight assigned to these channels involved in Equation 13.

Effects of Deformable(DRG) and Depth-wise Residual Group(DWRG).
We perform a series of ablation experiments by replacing DRG or/and DWRG with original RG, to demonstrate the effect of our modified RG structure. Respectively, in Spatial Extraction Network(SEN) we set num DRG = 3 and num RG = 7, in Frequency Reconstruction Network(FRN) we set num DW RG = 3 and num RG = 7. For each group, the number of residual blocks is set as 10. As shown in Table 3, the PSNR on Set5 increased by 0.18 dB when we replace specific RG with DRG, increased by 0.03 dB when we replace specific DWRG, and we can have the best performance by using both of them. The FRM on Set5 also increased when we replace RG with DRG and DWRG, by 0.27 and 0.05 respectively. The comparison shows the effectiveness of our proposed modified Residual Group architectures.

High-frequency Detail Enhancement based on other SR Models
As the output of our proposed model is a group of separated frequencydomain feature maps, we can easily merge our output with the output of other SR models, thus realize the enhancement on selected high-frequency channels. We first perform a similar process as 3.1 to convert the output from other SR model I SRori to its frequency-domain feature maps group F SRori , then we replace certain channels in F SRori with the corresponding channels in F output from FreqNet to get the merged output F merge .

Conclusions
We propose FreqNet, a frequency-domain image super-resolution model that explicitly learn the reconstruction of high-frequency details from LR images. We propose the depth-wise residual group(DWRG) and deformable residual group(DRG) structure to fit the characteristics of frequency-domain task and improve the ability of our network. Meanwhile, we propose a frequency-domain loss function and the frequency-domain reconstruction metric(FRM) that can measure the quality of high-frequency detail reconstruction. The quantitative and visual results demonstrate the effectiveness of our method, and we can further merge the output of our network with the other SR models as a post-processing enhancement.