{'fraction_non_alphanumeric': 0.04019313032061327, 'fraction_numerical': 0.014611833467451612, 'mean_word_length': 4.750608865075499, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Feedback Pyramid Attention Networks for Single Image Super-Resolution

Recently, convolutional neural network (CNN) based image super-resolution (SR) methods have achieved significant performance improvement. However, most CNN-based methods mainly focus on feed-forward architecture design and neglect to explore the feedback mechanism, which usually exists in the human visual system. In this paper, we propose feedback pyramid attention networks (FPAN) to fully exploit the mutual dependencies of features. Specifically, a novel feedback connection structure is developed to enhance low-level feature expression with high-level information. In our method, the output of each layer in the first stage is also used as the input of the corresponding layer in the next state to re-update the previous low-level filters. Moreover, we introduce a pyramid non-local structure to model global contextual information in different scales and improve the discriminative representation of the network. Extensive experimental results on various datasets demonstrate the superiority of our FPAN in comparison with the state-of-the-art SR methods.


I. INTRODUCTION
Single image super-resolution (SISR) [1] is a class of techniques that infers a high resolution (HR) image from its corresponding low resolution (LR) image. SR is an ill-posed inverse problem since each LR patch has to be mapped onto multiple HR patches. To solve this problem, researchers have proposed a multitude of learning-based methods to learn the mapping function from LR patches to their HR counterparts [2][3][4][5][6][7][8][9][10][11][12][13].
Due to the strong learning ability, deep neural network [14] based methods have been proposed to learn the nonlinear mapping between LR and HR image pairs. Dong et al. [4] firstly presented a super-resolution convolutional neural network (SRCNN) to learn an end-to-end nonlinear mappings from LR to its HR counterpart. Kim et al. proposed a very deep super-resolution (VDSR) network [5] by using residual learning with up to 20 convolutional layers, achieving substantial improvement over the SRCNN method. In order to reduce the parameters of the model, Kim [7], which utilized local and global residual learning to train a deeper model and showed favorable performances against DRCN with fewer parameters. Based on Laplacian pyramid framework, Lai et al. [8] proposed the Laplacian Pyramid Super-Resolution Network (LapSRN) to reconstruct HR images progressively. Ledig et al. [10] applied residual net (ResNet) architecture to construct a deeper network (SRResNet) for image SR. Enhanced Deep Super-Resolution (EDSR) [11] proposed by Lee et al. employed a simplified ResNet [15] architecture by removing the normalization layers in the SRResNet and won the championship of the NTIRE2017 super-resolution challenge [16]. To further improve the performance of the networks, Residual Dense Network (RDN) [17] and Channel-wise and Spatial Feature Modulation Network (CSFM) [18] utilize the residual and the dense skip connections [19] in their networks to fuse multilevel features for the SR reconstruction.
Although these feedforward methods achieve good results, the feedback mechanism [20][21][22][23] has not been fully exploited in deep learning based super-resolution methods. In cognition theory, feedback connections that link the cortical visual areas can transfer response signals from high-level cortical areas to the current low-level neuron [21,22]. Inspired by this mechanism, the feedback connections have been used in many deep neural networks [24][25][26]. The feedback connections can bring the semantic information from the higher layers back to refine the information of the previous layers in a top-down manner. Recently, Haris et al. [27] introduced the feedback mechanism into an SR module to update feature errors iteratively. Liu et al. [28] proposed the new feedback version Hierarchical Back Projection Network (HBPN), applying the new up-sampling and down-sampling back projection block to return the LR and HR residual errors. To further improve the performance, Liu et al. [29] introduced the attention mechanism to the feedback block to refine the back projection network. Latterly, Li et al. [30] employed the recurrent structure with feedback connections to correct the low-level representation information. CliqueNet constructed by Yang  Zhong et al. [32] introduced clique structures into superresolution and obtained a better result. However, these extreme complex dense connections can increase the training difficulty of the network. Therefore, in this paper, we propose a novel simple feedback connection structure to promote the flow of information through the network. Attention mechanism. In recent years, attention mechanisms have been successfully applied to a large number of tasks via rescaling the feature response to concentrate on the important components of the inputs [33][34][35][36]. Hu et al. [33] introduced the channel-wise attention mechanism, which proposed a squeeze-and-excitation block (SENet) to rescale channel-wise feature responses for image classification and achieved excellent results. Later, Zhang et al. [34] directly integrated the squeeze-and-excitation block into the residual architecture to construct a residual channel attention network (RCAN) and achieved significant performance gains for image SR. Inspired by classic non-local mean, Wang et al. [37] introduced a non-local neural network that firstly incorporated non-local operations in deep convolutional neural networks for image recognition and video classification. In the non-local block, the spatial attention masks are obtained by computing the similarity matrix between each spatial position in the input feature map, then they adaptively modulate feature representations at a position by a weighted sum of the features at all positions based on the obtained attention masks. Considering that the non-local network (NL) calculates the correlation on the entire feature map, which will increase the computational complexity dramatically when the spatial dimensionality is large. In order to resolve this problem, Liu et al. [38] introduced a non-local recurrent network (NLRN) with limited neighborhood for image restoration. Dai et al. [35] adopted region-level non-local enhancement operations in second-order attention network (SAN). However, SAN [35] and NLRN [38] still suffer from a huge computational burden, which greatly hinders the use of non-local blocks. GCNet proposed by Cao et al. [39] surprisingly found that the attention maps of different query positions are almost the same. By simplifying the nonlocal block, GCNet has the advantage of the non-local block with capturing the long-range dependencies and the SENet with lightweight property.
In this paper, we propose a novel feedback pyramid attention network (termed FPAN) for SISR, which is shown in Fig.  1. In the FPAN, we can not only enhance the representative ability of the proposed network but also improve the flow of information across the network by using a feedback connection structure. As a building block, feedback pyramid attention block (FPAB) is constructed by multiple forward skip connection layers and feedback skip connection layers, which can generate powerful high-level representations (illustrated in Fig. 2). The principle of our feedback scheme is that the output of each layer in the first stage is used as the input of the corresponding layer in the second stage to refine lowlevel representations. In each block, we further put forward to Laplacian pyramid attention to study non-local correlations at different scales, which is beneficial to enhance the discriminatory representation of the network (shown in Fig. 2 (b)). The corresponding experiments show that our proposed method achieved favorable results in comparison with the state-of-theart SR methods. In summary, the main contributions of our method are as follows.
(1) We introduce a feedback pyramid attention network (FPAN) for SISR. Our FPAN can adaptively refine feature representations and capture richer context information for image SR.
(2) We propose a novel feedback connection mechanism, which not only efficiently improves the information flow, but also enhances high-level feature representation via multiple skip connections. Besides, the proposed pyramid non-local block can capture long-distance spatial contextual information at multiple scales, which efficiently and adaptively recalibrates feature presentations. In comparison with state-of-theart methods, our method shows superiority in terms of model size and performance.
The rest of this paper is organized as follows. In Section II, we analyze the proposed FPAN in detail. Experimental results with analysis are presented in Section III to demonstrate the efficiency of our method. Finally, conclusions are provided in Section IV.

II. PROPOSED METHOD
In this section, we will introduce the proposed feedback pyramid attention networks in detail.  denotes matrix multiplication. ⊕ denotes broadcast element-wise addition.

A. Network Architecture
The overall framework of our FPAN is illustrated in Fig. 1, which is composed of two parts: 1) feature extraction part, and 2) reconstruction part. In this paper, I LR and I SR represent the input LR image and output HR image of our FPAN. We firstly utilize one convolution layer to extract the initial features F 0 from the input LR.
where H 0 denotes a convolution operation. F 0 is then fed into a series of feedback pyramid attention blocks (FPABs) to refine feature maps. Supposing G FPABs are used for further feature extraction, that is where F 0 is the output of the first convolution layer H 0 , After acquiring a set of feature maps [F G , F G−1 , · · · , F 1 ], we apply global feature fusion and global skip connection: where H hf f is composed of hierarchical feature fusion (1 × 1 convolution filter) and last feature extraction function (3 × 3 convolution filter), F GF is the output of the feature extraction part. Subsequently, we utilize a sub-pixel convolution layer [40] followed by a convolution layer for reconstructing HR image.
where H recon is the upscaling and reconstruction operation, H F P AN represents the function of our FPAN. The FPAN is optimized with the L 1 loss function [34][35][36] by minimizing the difference between the reconstructed image I SR and the ground-truth I HR . Given a training dataset , where N represents the number of the LR inputs and the corresponding HR inputs. The loss function is formulated as where θ denotes the parameter set of the FPAN. In the next subsection, we will introduce the proposed feedback pyramid attention block in detail.

B. Feedback Pyramid Attention Block (FPAB)
The structure of feedback pyramid attention block (FPAB) is shown in Fig. 2, the module mainly contains three parts: feedback connection structure, pyramid non-local block and local residual learning. Let F g−1 and F g denote the input and output of the g-th FPAB and F g can be obtained by where f F C and f P N are the proposed feedback connection structure and pyramid non-local block. 1) Feedback Connection Structure: Feedback mechanism is known as the efficient iterative procedure to refine lowlevel representations using high-level semantic information. Unlike the clique structure [31] that each layer is both the input and output of another one in the same block, we propose a novel feedback connection mechanism, where the output of each layer except the input layer in the first stage is used as the input of the corresponding layer in the second stage in the same block. As illustrated in Fig. 2 (a), the proposed feedback structure contains two stages. At the beginning, the input layer F g−1 initializes all layers by simple feedforward skip connections. The outputs of two adjacent convolutional layers are fed to the subsequent layer in the first stage. The second stage begins to update the feature maps by building a skip connection between the corresponding layers of the two stages. Specifically, the proposed feedback connection mechanism can be presented as follows. The initial stage is The first stage is and the second stage is where X i j denotes the feature maps of the i-th (i ≥ 1) convolutional layer in the j-th (j = 1, 2) stage. X −1 1 =F g−1 , X 0 1 = X 0 . X 0 2 =X 1 is the output of the first stage. * is the convolutional operator. σ represents the activation function (ReLU) [41]. [·, ·] refers to the concatenation operation. Then, the output of each stage is fed to the subsequent layers to enhance the information propagation in the network.
2) Pyramid Non-local Block: The aim of SR is to restore high frequency components of images. However, most SR methods recover image details blindly and lack abilities to identify high frequency regions. Thus, we consider if we can enhance the network sensitivity to high frequency information, thereby further improving the representational ability of the network, which is critical for SR. On the other hand, the convolution operation focuses only on local regions and cannot obtain long-dependencies at a time. In view of this, in this paper, we design a pyramid non-local block illustrated in Fig.  2 (b), which can capture rich contextual information in the spatial dimensions and improve the discriminative ability of the model. a) Non-local Module: The non-local module [37] can learn long-range relationships among any spatial positions and has been widely applied in visual tasks. In order to further improve the representation ability of the network, we will introduce a non-local module into the network to capture rich contextual information and enlarge the receptive field to the entire feature map. Let X = {x i } N i=1 denotes the input of the non-local module, which has C feature maps with size of H × W . For the convenience of the calculation, we reshape X to X ∈ R C×N , i.e., N = H × W and the spatial attention operation is defined as where S ∈ R N ×N , θ (·) and ϕ (·) are two linear feature embeddings (e.g. 1 × 1 convolution operations). f (·, ·) is the similarity function that measures the correlations between any two positions in the embedding feature map. Specifically, most papers adopt softmax as a measure function: where s ij represents the similarity between the j th location x j and the i th location x i in the spatial dimension. Similarly, we also consider a linear embedding operation for input X: g (X) = W g X with the learned parameters W g , which can be implemented as 1×1 convolution and we reshape it to R C×N . Then, the final output y i at location i as following: where j is the index that enumerates all possible positions, capturing long-range contextual correlations across the whole feature map. W y is the learned weight parameter (initialized as zero) and the residual connection structure "+x i " guarantee the flow of the original information. However, this non-local operation requires a lot of computational costs, especially when the spatial dimensionality is large. A global context block (GC) was presented in [39]. The GC block simplifies the computational complexity of non-local module with SE block [33], while enhancing the discriminative presentation of the network based on an independent attention map. The GC block mainly contains three steps, 1) adopt global attention pooling to implement context modeling, 2) transform operation to achieve channel-wise dependencies, 3) where W k , W v1 , and W v2 denote three linear feature embeddings (e.g. 1 × 1 convolution operations), α j = e W k x j N m=1 e W k xm represents the weight for context modeling, LN (·) is layer normalization [42], δ (·) = W v2 ReLU (LN (W v1 (·))) is the feature transform, and "+" denotes the feature fusion.
b) Pyramid Non-local Mechanism: To make full use of non-local power, pyramid attention is introduced to boost and exploit the relationship between the features at different scales. The corresponding architecture is illustrated in Fig. 2 where P N LB (·) denotes pyramid non-local operation, i is the scale parameter of the Laplacian pyramid, P A i represents the context modeling module at the i-th scale. Concat (·) is a feature concatenation operation. The lightweight of the PNLB allows it to be plugged into multiple layers to better capture the long-range dependency with only a slight computational burden.

B. Implementation Details
Data augmentation is performed on above 800 training images, which includes random rotating 90 • , 180 • , 270 • and horizontal flipping. In each min-batch, 16 LR color patches with the size of 48 × 48 are randomly extracted as inputs. Our model is optimized with Adam [49], where the parameters are set to β 1 = 0.9, β 2 = 0.999, and ε= 10 −8 . The learning rate is initialized to 10 −4 and reduced to half every 200 epochs. We implement our model by utilizing PyTorch [50] with an Nvidia 2080Ti GPU.
In the proposed network, we set all convolution layers to have 64 filters and the kernel size is 3 × 3 except for those 1 × 1 convolutional layers in the pyramid non-local unit and feature fusion part. We adopt zero-padding strategy to keep the feature size fixed. In PNLB, we use 6 × 6 convolutional layer with two striding and two padding to perform downsampling operations. In addition, the reduction ratio of the 1 × 1 convolutional layer is set to 16 in the pyramid nonlocal unit. In each feedback connection block, the number of convolutional layers is 8. For upscaling module, we follow [11,34] and use ESPCNN [40] to perform the upsampling operation. Meanwhile, the final convolution layer with 3 filters is used to produce the desired HR image.

C. Ablation Analysis
In this section, we investigate the effects of different components, including feedforward dense skip connections, feedback skip connections and pyramid attention. For a fair comparison, the baseline and its variants share the same experimental configuration and are trained from scratch.
To demonstrate the effectiveness of the proposed feedback connection mechanism, we first compare the feedback network (P 2 ) with its feedforward counterpart (P 0 , P 1 ). As shown in Table I, the baseline P 0 (without the feedforward dense skip connections, the feedback skip connections and attention mechanism) performance is relatively low, its PSNR only reaches 32.33 dB on Set5 (×4). Then, P 1 that adds the feedforward dense skip connections obtain 32.35 dB. After integrating the feedback skip connections, the performance can be improved from 32.35 dB to 32.43 dB. Meanwhile, as discussed above, we further show the effect of attention. It can be seen that the network with attention (P 3 and P 4 ) perform better than those without attention (from P 0 to P 2 ). These observations indicate that the attention mechanism can adaptively exploit more important information for SR. Specifically, when we compare the global context block [ (P 3 , S = 1) and pyramid non-local block (P 4 , S = 1, 2, 4), we can see that our pyramid attention mechanism can achieve better performance than the network with the global context block (i.e., 32.55 dB v.s. 32.49 dB). In addition, we also show some other quantitative comparison results, including: the number of parameters and the number of floating-point operations (FLOPs), which are illustrated in Fig. 3. FLOPs is computed with 512×512 HR image at ×4 scaling factor. It can be find that although more pyramid scales can improve PSNR, the number of parameters and PLOPs will also increase. To achieve a better tradeoff, we set the pyramid scale to 1, 2, 4. These comparisons consistently demonstrate the superiority of our proposed FPAN.

D. Results with Bicubic Degradation (BI)
To verify the effectiveness of our FPAN, we compare our FPAN with some other state-of-the-art SR methods: Bicubic, SRCNN [4], VDSR [5], DRCN [6], LapSRN [8], EDSR [11], DBPN [27] and SRFBN [30]. As in [11,17,34], we also adopt self-ensemble strategy to further improve our FPAN and denote the self-ensembled FPAN as FPAN+. Table II shows all the quantitative results for various scale factors. In comparison with these methods, our FPAN+ achieves the best performance on all the datasets with various scale factors. Without selfensemble, our FPAN also obtains comparable performance and outperforms almost all the other methods. Although there is no absolute advantage compared to some state-of-the-art methods, our FPAN has fewer layers and parameters than EDSR [11] (11.7 M v.s. 43 M) and employs fewer training images than DBPN [27] and SRFBN [30] (DIV2K v.s. DIV2K + Flickr2K + ImageNet v.s. DIV2K + Flickr2K).
To further illustrate the superiority of our FPAN, in Fig. 4, we show visual results with an upscaling factor ×4 for these images from Urban100 dataset. From that, we can see that our method yields the best visual results among all existing compared methods. Using "image 092" as an example, we can see that the Bicubic interpolation generates heavy blurring artifacts along the edges and visually displeasing textures. Later, some developed methods can recover some structure, but still suffer from the wrong texture direction and fail to recover more accurate image details. However, the proposed FPAN alleviates distorted lines and produces more faithful textures and structures. The reason is that FPAN can use highlevel information to refine the low-level feature information to achieve self-correction. In addition, pyramid attention mechanism is used to further enhance the discriminative representation of the network, thus generating a better SR result.

E. Results with BD and DN Degradations
Following [17,30], we also show SR results with BD and DN degradation models. We compare our FPAN with some state-of-the-art methods: SRCNN [4], FSRCNN [51], VDSR [5], "IRCNN G" [52], "IRCNN C" [52], RDN [17], SRFBN [30]. As shown in Table III, our FPAN and FPAN+ generate better performance than other methods on almost all quantitative results with scaling factor ×3. In particular, for the BD degradation model, our method achieves better performance than other state-of-the-art methods even without self-ensemble.
In Fig. 5, we show visual comparisons with BD and DN degradation models. The recovered results from other methods are blurry. In contrast, our FPAN and FPAN+ can alleviate the distortion and produce more faithful details in SR results. It also verifies that our FPAN has the ability to capture the important discriminative features for BD and DN degradation models.

F. Model Size Comparison
In Fig. 6, we illustrate the comparisons about model size and performance on Set 5 with scale factor ×2. Our FPAN has much fewer parameters than EDSR, but obtains higher performance. In comparison with DBPN and SRFBN, our FPAN and FPAN+ obtain better results with a relatively large model. It implies that our FPAN has a good tradeoff between model size and performance.

IV. CONCLUSION
In this paper, we propose a deep feedback pyramid attention networks (FPAN) for image SR. Specially, the proposed feedback connection structure allows FPAN to not only refine low-level representation with high-level information, but also effectively boost the flow of information as well as the feature reuse. In addition, we introduce a pyramid non-local block to capture the long-distance dependencies in multiple scales and make the network concentrate on learning important highfrequency information. Extensive evaluations on SR with BI, BD and DN models show the superior performance of our FPAN in terms of quantitative and visual results.