{'fraction_non_alphanumeric': 0.041413753361505955, 'fraction_numerical': 0.019631194775259316, 'mean_word_length': 5.161183431952662, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Restormer: Efficient Transformer for High-Resolution Image Restoration

Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising). The source code and pre-trained models are available at https://github.com/swz30/Restormer.


Introduction
Image restoration is the task of reconstructing a highquality image by removing degradations (e.g., noise, blur, rain drops) from a degraded input. Due to the ill-posed nature, it is a highly challenging problem that usually requires strong image priors for effective restoration. Since convolutional neural networks (CNNs) perform well at learning generalizable priors from large-scale data, they have emerged as a preferable choice compared to conventional restoration approaches.
The basic operation in CNNs is the 'convolution' that provides local connectivity and translation equivariance. While these properties bring efficiency and generalization to CNNs, they also cause two main issues.

DBGAN
(CVPR20) [94] [58] [14] [13] [93] [100] MPRNet (CVPR21) [93] [64] [43] [32] (a) Deblurring   DeamNet (CVPR21) [63] [55] [80] [93] [92] (c) Gaussian Denoising (Tab. 4) (d) Real Denoising (Tab. 6) lution operator has a limited receptive field, thus preventing it from modeling long-range pixel dependencies. (b) The convolution filters have static weights at inference, and thereby cannot flexibly adapt to the input content. To deal with the above-mentioned shortcomings, a more powerful and dynamic alternative is the self-attention (SA) mechanism [17,77,79,95] that calculates response at a given pixel by a weighted sum of all other positions. Self-attention is a core component in Transformer models [34,77] but with a unique implementation, i.e., multihead SA that is optimized for parallelization and effective representation learning. Transformers have shown state-ofthe-art performance on natural language tasks [10,19,49,62] and on high-level vision problems [11,17,76,78]. Although SA is highly effective in capturing long-range pixel interactions, its complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to highresolution images (a frequent case in image restoration). Recently, few efforts have been made to tailor Transformers for image restoration tasks [13,44,80]. To reduce the computational loads, these methods either apply SA on small spatial windows of size 8×8 around each pixel [44,80], or divide the input image into non-overlapping patches of size 48×48 and compute SA on each patch independently [13]. However, restricting the spatial extent of SA is contradictory to the goal of capturing the true long-range pixel relationships, especially on high-resolution images.
In this paper, we propose an efficient Transformer for image restoration that is capable of modeling global connectivity and is still applicable to large images. Specifically, we introduce a multi-Dconv head 'transposed' attention (MDTA) block (Sec. 3.1) in place of vanilla multi-head SA [77], that has linear complexity. It applies SA across feature dimension rather than the spatial dimension, i.e., instead of explicitly modeling pairwise pixel interactions, MDTA computes cross-covariance across feature channels to obtain attention map from the (key and query projected) input features. An important feature of our MDTA block is the local context mixing before feature covariance computation. This is achieved via pixel-wise aggregation of crosschannel context using 1×1 convolution and channel-wise aggregation of local context using efficient depth-wise convolutions. This strategy provides two key advantages. First, it emphasizes on the spatially local context and brings in the complimentary strength of convolution operation within our pipeline. Second, it ensures that the contextualized global relationships between pixels are implicitly modeled while computing covariance-based attention maps.
A feed-forward network (FN) is the other building block of the Transformer model [77], which consists of two fully connected layers with a non-linearity in between. In this work, we reformulate the first linear transformation layer of the regular FN [77] with a gating mechanism [16] to improve the information flow through the network. This gating layer is designed as the element-wise product of two linear projection layers, one of which is activated with the GELU non-linearity [27]. Our gated-Dconv FN (GDFN) (Sec. 3.2) is also based on local content mixing similar to the MDTA module to equally emphasize on the spatial context. The gating mechanism in GDFN controls which complementary features should flow forward and allows subsequent layers in the network hierarchy to specifically focus on more refined image attributes, thus leading to high-quality outputs.
Apart from the above architectural novelties, we show the effectiveness of our progressive learning strategy for Restormer (Sec. 3.3). In this process, the network is trained on small patches and large batches in early epochs, and on gradually large image patches and small batches in later epochs. This training strategy helps Restormer to learn context from large images, and subsequently provides quality performance improvements at test time. We conduct comprehensive experiments and demonstrate state-of-theart performance of our Restormer on 16 benchmark datasets for several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (on single-image and dual pixel data), and image denoising (on synthetic and real data); See Fig. 1. Furthermore, we provide extensive ablations to show the effectiveness of architectural designs and experimental choices. The main contributions of this work are summarized below: • We propose Restormer, an encoder-decoder Transformer for multi-scale local-global representation learning on high-resolution images without disintegrating them into local windows, thereby exploiting distant image context. • We propose a multi-Dconv head transposed attention (MDTA) module that is capable of aggregating local and non-local pixel interactions, and is efficient enough to process high-resolution images. • A new gated-Dconv feed-forward network (GDFN) that performs controlled feature transformation, i.e., suppressing less informative features, and allowing only the useful information to pass further through the network hierarchy.

Background
Image Restoration. In recent years, data-driven CNN architectures [7,18,92,93,105,107] have been shown to outperform conventional restoration approaches [26,36,53,75]. Among convolutional designs, encoder-decoder based U-Net architectures [3,14,39,80,90,93,99] have been predominantly studied for restoration due to their hierarchical multi-scale representation while remaining computationally efficient. Similarly, skip connection based approaches have been shown to be effective for restoration due to specific focus on learning residual signals [24,48,92,106]. Spatial and channel attention modules have also been incorporated to selectively attend to relevant information [43,92,93]. We refer the reader to NTIRE challenge reports [2,5,30,57] and recent literature reviews [8,42,73], which summarize major design choices for image restoration. Vision Transformers. The Transformer model is first developed for sequence processing in natural language tasks [77]. It has been adapted in numerous vision tasks such as image recognition [17,76,88], segmentation [78,83,108], object detection [11,50,109]. The Vision Transformers [17,76] decompose an image into a sequence of patches (local windows) and learn their mutual relationships. The distinguishing feature of these models is the strong capability to learn long-range dependencies between image patch sequences and adaptability to given input content [34]. Due to these characteristics, Transformer models have also been studied for the low-level vision problems such as super-resolution [44,85], image colorization [37], denoising [13,80], and deraining [80]. However, the computational complexity of SA in Transformers can increase quadratically with the number of image patches, therby prohibiting its application to high-resolution images. Therefore, in lowlevel image processing applications, where high-resolution  outputs need to be generated, recent methods generally employ different strategies to reduce complexity. One potential remedy is to apply self-attention within local image regions [44,80] using the Swin Transformer design [44]. However, this design choice restricts the context aggregation within local neighbourhood, defying the main motivation of using self-attention over convolutions, thus not ideally suited for image-restoration tasks. In contrast, we present a Transformer model that can learn long-range dependencies while remaining computationally efficient.

Method
Our main goal is to develop an efficient Transformer model that can handle high-resolution images for restoration tasks. To alleviate the computational bottleneck, we introduce key designs to the multi-head SA layer and a multiscale hierarchical module that has lesser computing requirements than a single-scale network [44]. We first present the overall pipeline of our Restormer architecture (see Fig. 2). Then we describe the core components of the proposed Transformer block: (a) multi-Dconv head transposed attention (MDTA) and (b) gated-Dconv feed-forward network (GDFN). Finally, we provide details on the progressive training scheme for effectively learning image statistics. Overall Pipeline. Given a degraded image I ∈ R H×W ×3 , Restormer first applies a convolution to obtain low-level feature embeddings F 0 ∈ R H×W ×C ; where H ×W denotes the spatial dimension and C is the number of channels. Next, these shallow features F 0 pass through a 4-level symmetric encoder-decoder and transformed into deep features F d ∈ R H×W ×2C . Each level of encoder-decoder contains multiple Transformer blocks, where the number of blocks are gradually increased from the top to bottom levels to maintain efficiency. Starting from the high-resolution input, the encoder hierarchically reduces spatial size, while expanding channel capacity. The decoder takes low-resolution latent features F l ∈ R H 8 × W 8 ×8C as input and progressively recovers the high-resolution representations. For feature downsampling and upsampling, we apply pixel-unshuffle and pixel-shuffle operations [69], respectively. To assist the recovery process, the encoder features are concatenated with the decoder features via skip connections [66]. The concatenation operation is followed by a 1×1 convolution to reduce channels (by half) at all levels, except the top one. At level-1, we let Transformer blocks to aggregate the low-level image features of the encoder with the high-level features of the decoder. It is beneficial in preserving the fine structural and textural details in the restored images. Next, the deep features F d are further enriched in the refinement stage operating at high spatial resolution. These design choices yield quality improvements as we shall see in the experiment section (Sec. 4). Finally, a convolution layer is applied to the refined features to generate residual image R ∈ R H×W ×3 to which degraded image is added to obtain the restored image:Î = I + R. Next, we present the modules of the Transformer block.

Multi-Dconv Head Transposed Attention
The major computational overhead in Transformers comes from the self-attention layer. In conventional SA [17,77], the time and memory complexity of the key-query dotproduct interaction grows quadratically with the spatial resolution of input, i.e., O(W 2 H 2 ) for images of W ×H pixels. Therefore, it is infeasible to apply SA on most image restoration tasks that often involve high-resolution im-ages. To alleviate this issue, we propose MDTA, shown in Fig. 2(a), that has linear complexity. The key ingredient is to apply SA across channels rather than the spatial dimension, i.e., to compute cross-covariance across channels to generate an attention map encoding the global context implicitly. As another essential component in MDTA, we introduce depth-wise convolutions to emphasize on the local context before computing feature covariance to produce the global attention map.
From a layer normalized tensor Y ∈ RĤ ×Ŵ ×Ĉ , our MDTA first generates query (Q), key (K) and value (V) projections, enriched with local context. It is achieved by applying 1×1 convolutions to aggregate pixel-wise cross-channel context followed by 3×3 depth-wise convolutions to encode channel-wise spatial context, yielding d is the 3×3 depth-wise convolution. We use bias-free convolutional layers in the network. Next, we reshape query and key projections such that their dot-product interaction generates a transposed-attention map A of size RĈ ×Ĉ , instead of the huge regular attention map of size RĤŴ ×ĤŴ [17,77]. Overall, the MDTA process is defined as: where X andX are the input and output feature maps; Q ∈ RĤŴ ×Ĉ ;K ∈ RĈ ×ĤŴ ; andV ∈ RĤŴ ×Ĉ matrices are obtained after reshaping tensors from the original size RĤ ×Ŵ ×Ĉ . Here, α is a learnable scaling parameter to control the magnitude of the dot product ofK andQ before applying the softmax function. Similar to the conventional multi-head SA [17], we divide the number of channels into 'heads' and learn separate attention maps in parallel.

Gated-Dconv Feed-Forward Network
To transform features, the regular feed-forward network (FN) [17,77] operates on each pixel location separately and identically. It uses two 1×1 convolutions, one to expand the feature channels (usually by factor γ=4) and second to reduce channels back to the original input dimension. A non-linearity is applied in the hidden layer. In this work, we propose two fundamental modifications in FN to improve representation learning: (1) gating mechanism, and (2) depthwise convolutions. The architecture of our GDFN is shown in Fig. 2(b). The gating mechanism is formulated as the element-wise product of two parallel paths of linear transformation layers, one of which is activated with the GELU non-linearity [27]. As in MDTA, we also include depth-wise convolutions in GDFN to encode information from spatially neighboring pixel positions, useful for learning local image structure for effective restoration. Given an input tensor X ∈ RĤ ×Ŵ ×Ĉ , GDFN is formulated as: where denotes element-wise multiplication, φ represents the GELU non-linearity, and LN is the layer normalization [9]. Overall, the GDFN controls the information flow through the respective hierarchical levels in our pipeline, thereby allowing each level to focus on the fine details complimentary to the other levels. That is, GDFN offers a distinct role compared to MDTA (focused on enriching features with contextual information). Since the proposed GDFN performs more operations as compared to the regular FN [17], we reduce the expansion ratio γ so as to have similar parameters and compute burden.

Progressive Learning
CNN-based restoration models are usually trained on fixed-size image patches. However, training a Transformer model on small cropped patches may not encode the global image statistics, thereby providing suboptimal performance on full-resolution images at test time. To this end, we perform progressive learning where the network is trained on smaller image patches in the early epochs and on gradually larger patches in the later training epochs. The model trained on mixed-size patches via progressive learning shows enhanced performance at test time where images can be of different resolutions (a common case in image restoration). The progressive learning strategy behaves in a similar fashion to the curriculum learning process where the network starts with a simpler task and gradually moves to learning a more complex one (where the preservation of fine image structure/textures is required). Since training on large patches comes at the cost of longer time, we reduce the batch size as the patch size increases to maintain a similar time per optimization step as of the fixed patch training.

Experiments and Analysis
We evaluate the proposed Restormer on benchmark datasets and experimental settings for four image processing tasks: (a) image deraining, (b) single-image motion deblurring, (c) defocus deblurring (on single-image, and dualpixel data), and (d) image denoising (on synthetic and real data). More details on datasets, training protocols, and additional visual results are presented in the supplementary material. In tables, the best and second-best quality scores of the evaluated methods are highlighted and underlined. Implementation Details. We train separate models for different image restoration tasks. In all experiments, we use the following training parameters, unless mentioned otherwise. Our Restormer employs a 4-level encoder-decoder.    [1,2,4,8], and number of channels are [48,96,192,384]. The refinement stage contains 4 blocks. The channel expansion factor in GDFN is γ=2.66. We train models with AdamW optimizer (β 1 =0.9, β 2 =0.999, weight decay 1e −4 ) and L 1 loss for 300K iterations with the initial learning rate 3e −4 gradually reduced to 1e −6 with the cosine annealing [51]. For progressive learning, we start training with patch size 128×128

Image Deraining Results
We compute PSNR/SSIM scores using the Y channel in YCbCr color space in a way similar to existing methods [32,61,93]. Table 1 shows that our Restormer achieves consistent and significant performance gains over existing approaches on all five datasets. Compared to the recent best method SPAIR [61], Restormer achieves 1.05 dB improvement when averaged across all datasets. On individual datasets, the gain can be as large as 2.06 dB, e.g., Rain100L. Figure 3 shows a challenging visual example. Our Restormer reproduces a raindrop-free image while effectively preserving the structural content.

Single-image Motion Deblurring Results
We evaluate deblurring methods both on the synthetic datasets (GoPro [56], HIDE [67]) and the real-world datasets (RealBlur-R [65], RealBlur-J [65]). Table 2 shows that our Restormer outperforms other approaches on all four benchmark datasets. When averaged across all datasets, our method obtains a performance boost of 0.47 dB over the recent algorithm MIMO-UNet+ [14] and 0.26 dB over the previous best method MPRNet [93]. Compared to MPR-Net [93], Restormer has 81% fewer FLOPs (See Fig. 1 Figure 5. Dual-pixel defocus deblurring comparison on the DPDD dataset [3]. Compared to the other approaches, our Restormer more effectively removes blur while preserving the fine image details. Moreover, our method shows 0.4 dB improvement over the Transformer model IPT [13], while having 4.4× fewer parameters and runs 29× faster. Notably, our Restormer is trained only on the GoPro [56] dataset, yet it demonstrates strong generalization to other datasets by setting new stateof-the-art. Fig. 4 shows that the image produced by our method is more sharper and visually closer to the groundtruth than those of the other algorithms. Table 3 shows image fidelity scores of the conventional defocus deblurring methods (EBDB [33] and JNB [68]) as well as learning based approaches on the DPDD dataset [3]. Our Restormer significantly outperforms the state-of-the-art schemes for the single-image and dual-pixel defocus deblurring tasks on all scene categories. Particularly on the combined scene category, Restormer yields ∼ 0.6 dB improvements over the previous best method IFAN [41]. Compared to the Transformer model Uformer [80], our method provides a substantial gain of 1.01 dB PSNR. Figure 5 illustrates that our method is more effective in removing spatially varying defocus blur than other approaches.
Gaussian denoising. Table 4 and Table 5 show PSNR scores of different approaches on several benchmark datasets for grayscale and color image denoising, respectively. Consistent with existing methods [44,99], we include noise levels 15, 25 and 50 in testing. The evaluated methods are divided into two experimental categories: (1) learning a single model to handle various noise levels, and (2) learning a separate model for each noise level. Our Restormer achieves state-of-the-art performance under both experimental settings on different datasets and noise levels. Specifically, for the challenging noise level 50 on high-resolution Urban100 dataset [29], Restormer achieves 0.37 dB gain over the previous best CNN-based method DRUNet [99], and 0.31 dB boost over the recent Table 7.
Ablation experiments for the Transformer block. PSNR is computed on a high-resolution Urban100 dataset [29]. transformer-based network SwinIR [44], as shown in Table 4. Similar performance gains can be observed for the Gaussian color denoising in Table 5. It is worth mentioning that DRUNet [99] requires the noise level map as an additional input, whereas our method only takes the noisy image. Furthermore, compared to SwinIR [44], our Restormer has 3.14× fewer FLOPs and runs 13× faster. Figure 6 presents denoised results by different methods for grayscale denoising (top row) and color denoising (middle row). Our Restormer restores clean and crisp images. Real image denoising. Table 6 shows that our method is the only one surpassing 40 dB PSNR on both datasets. Notably, on the SIDD dataset our Restormer obtains PSNR gains of 0.3 dB and 0.25 dB over the previous best CNN method MIRNet [92] and Transformer model Uformer [80], respectively. Fig. 6 (bottom row) shows that our Restormer generates clean image without compromising fine texture.

Ablation Studies
For ablation experiments, we train Gaussian color denoising models on image patches of size 128×128 for 100K iterations only. Testing is performed on Urban100 [29], and analysis is provided for a challenging noise level σ=50. FLOPs and inference time are computed on image size 256×256. Table 7-10 show that our contributions yield quality performance improvements. Next, we describe the influence of each component individually. Improvements in multi-head attention. Table 7c demonstrates that our MDTA provides favorable gain of 0.32 dB over the baseline (Table 7a). Furthermore, bringing locality to MDTA via depth-wise convolution improves robustness as removing it results in PSNR drop (see Table 7b).
Improvements in feed-forward network (FN). Table 7d shows that the gating mechanism in FN to control information flow yields 0.12 dB gain over the conventional FN [77]. As in multi-head attention, introducing local mechanism to FN also brings performance advantages (see Table 7e). We further strengthen the FN by incorporating gated depth-wise convolutions. Our GDFN (Table 7f) achieves PSNR gain of 0.26 dB over the standard FN [77] for the noise level 50. Overall, our Transformer block contributions lead to a significant gain of 0.51 dB over the baseline. Design choices for decoder at level-1. To aggregate encoder features with the decoder at level-1, we do not employ 1×1 convolution (that reduces channels by half) after concatenation operation. It is helpful in preserving fine textural details coming from the encoder, as shown in Table 8. These results further demonstrate the effectiveness of adding Transformer blocks in the refinement stage. Impact of progressive learning. Table 9 shows that the progressive learning provides better results than the fixed patch training, while having similar training time. Deeper or wider Restormer? Table 10 shows that, under similar parameters/FLOPs budget, a deep-narrow model performs more accurately than its wide-shallow counterpart. However, the wider model runs faster due to parallelization. In this paper we use deep-narrow Restormer.

Conclusion
We present an image restoration Transformer model, Restormer, that is computationally efficient to handle highresolution images. We introduce key designs to the core components of the Transformer block for improved feature aggregation and transformation. Specifically, our multi-Dconv head transposed attention (MDTA) module implicitly models global context by applying self-attention across channels rather than the spatial dimension, thus having linear complexity rather than quadratic. Furthermore, the proposed gated-Dconv feed-forward network (GDFN) introduces a gating mechanism to perform controlled feature transformation. To incorporate the strength of CNNs into the Transformer model, both MDTA and GDFN modules include depth-wise convolutions for encoding spatially local context. Extensive experiments on 16 benchmark datasets demonstrate that Restormer achieves the state-ofthe-art performance for numerous image restoration tasks. Table 8. Influence of concat (w/o 1x1 conv) and refinement stage at decoder (level-1). We add the components to experiment Table 7(g).