{'fraction_non_alphanumeric': 0.04118809228811573, 'fraction_numerical': 0.02163212001473395, 'mean_word_length': 4.77183996907615, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors

The image inpainting task fills missing areas of a corrupted image. Despite impressive results have been achieved recently, it is still challenging to restore corrupted images with both vivid textures and reasonable structures. Some previous methods only tackle regular textures while losing holistic structures limited by receptive fields of Convolution Neural Networks (CNNs). To this end, we study learning a Zero-initialized residual addition based Incremental Transformer on Structural priors (ZITS++), an improved model over our conference ZITS model. Specifically, given one corrupt image, we present the Transformer Structure Restorer (TSR) module to restore holistic structural priors at low image resolution, which are further upsampled by Simple Structure Upsampler (SSU) module to higher image resolution. Further, to well recover image texture details, we take the Fourier CNN Texture Restoration (FTR) module, which has both the Fourier and large-kernel attention convolutions. Typically, FTR can be independently pre-trained without image structural priors. Furthermore, to enhance the FTR, the upsampled structural priors from TSR are further processed by Structure Feature Encoder (SFE), and updating the FTR by a novel incremental training strategy of Zero-initialized Residual Addition (ZeroRA). Essentially, a new masking positional encoding is proposed to encode the large irregular masks. Extensive experiments on various datasets validate the efficacy of our model compared with other competitors. We also conduct extensive ablation to compare and verify various priors for image inpainting tasks.


INTRODUCTION
As a long-standing challenge, image inpainting aims at filling plausible pixels in missing areas of pictures, and thus is very useful to various real-world applications, such as object removal [3], photo restoration, and image editing [4]. To achieve realistic outcomes, the inpainted images should remain both semantically coherent textures in the low-level color space and visually reasonable structures for high-level human perception.
Many classical algorithms [5], [6], [7], [8], [9] heuristically search similar patches for the reconstruction of missing image regions, while it is still very non-trivial for these methods to preserve good textures and holistic structures of the corrupted image. Benefiting from the excellent capacities of CNNs [10] and Generative Adversarial Networks (GANs) [11], existing deep learning methods [2], [12], [13], [14], [15] could conduct the image inpainting tasks in some common cases. However, the capability to inpaint realistic image content is often limited in practice by the underlying challenges as follows.
(1) Limited receptive fields and position information. Learning semantically consistent textures is difficult for vanilla CNNs, as they only have the local inductive priors and narrow receptive field of convolutional filters. Even the dilated convolutions [16] are still incapable of tackling large corrupted regions in the images. Even worse, without explicitly encoding the positional information of corrupted image areas, the model is quite likely to just produce repeated and meaningless artifacts to fill large irregular masked regions. ⋆ indicates equal contributions, † refers to the corresponding author. (2) Missing holistic structures. As illustrated in Fig. 1(f), CNNs tend to copy nearby textures rather than maintain the global structure with the growth of image resolution. Thus without a holistic understanding of the images, it is difficult to recover key boundaries, lines and edges for scenes, especially for the images with weak texture.
(3) Heavy computations. Generally, it is very tricky and costly to training sophisticated GANs with large image sizes [17]. And the inpainting performance may be degraded on High-Resolution (HR) images. For example, it takes 2 weeks of training LaMa [2] from the scratch on 256×256 Places2 with 3 V100 16 GB GPUs for a total of 0.8 million steps. Thus it will take much more computational resources and time to train LaMa [2] on 512 × 512 Place2 dataset.
Some pioneering works can partially solve these problems. For the limited receptive fields, the attention mechanism is leveraged in [15], [18], [19]. Fast Fourier Convolution (FFC) of global receptive fields is utilized in [2] to encode features in the frequency domain for resolution-robust inpainting, but the holistic structure is not explicitly modeled in [2]. Some transformer-based methods [14], [20] with longrange dependency firstly fill Low-Resolution (LR) tokens, and then upsample them with CNNs to address the huge memory footprint of transformers on large images. On the other hand, various prior information of the images has been explored for inpainting, such as canny edges [12], [21], segmentation [22], [23], Relative Total Variation (RTV) [24], [25], learning based edges [26], gradients [27], and even semantic prior features [28]. Critically, our previous work [13] proposes the concept of sketch tensor space, and exploits the structural priors of edges and wireframes [29] in the space to facilitate holistic structure recovery for image inpainting. To sum up, these methods typically have the multi-stage or   [1] and ZITS++ enhanced with canny and CATS respectively. Right (f): High-resolution (1024×768, 1024×1024, 680×1024) inpainted results. Each image group: original image with inpainting mask, inpainted results by LaMa [2], and results of ZITS++. Please zoom-in for details.
multi-model designs, and are very expensive to be trained from the scratch. Essentially, it is also non-trivial to directly enable amenable structural priors to enhance a well pretrained inpainting model. Additionally, there is relative less study and discussion of using position information in image inpainting, while such position information is quite critical to the inpainting model [30], [31].
Therefore, we systematically address these problems. Particularly, Learning-based Edges (L-Edges) are advocated as the important representations for meaningful holistic structures. A novel transformer-based model is presented to restore holistic structures by L-Edges, wireframe lines and gradients. Critically, we further iteratively upsample to arbitrary resolution the grayscale L-Edges by Edge Non-Maximum Suppression (E-NMS), and lines by a simple CNN, respectively. This greatly benefits the structural prior learned in LR images, adaptive to the inpainting in HR images. A novel and well designed positional encoding scheme for the mask region is utilized to improve the performance of image restoration. Further, Fourier convolutions from [2] have been employed as the backbone for texture restoration. We significantly improve it with Large Kernel Attention (LKA) modules [32], as our FTR module. Typically, FTR can be independently pre-trained as in [2]. Furthermore, to reduce the computational cost, we propose a novel incrementally training strategy with Zero-initialized Residual Addition (ZeroRA) [33] to incorporate structural prior into a pre-trained FTR module. This incremental strategy enjoys fast convergence for much fewer steps compared with retraining a model from the scratch.
Formally, built upon our conference version [1], this paper proposes an improved ZeroRA based Incremental Transformer Structure (ZITS++) inpainting framework. Our ZITS++ has novel components of Transformer Structure Restorer (TSR), Simple Structure Upsampler (SSU), Masking Positional Encoding (MPE), Structure Feature Encoder (SFE), and Fourier CNN Texture Restoration (FTR). Given the corrupted image, ZITS++ has this workflow: TSR first recovers the structural priors in 256×256 of the input image. We upsample these priors by SSU and further take them into SFE for structural features, which are finally leveraged by the CNN-based FTR for restoring textures trained by the ZeroRA strategy. Specifically, the TSR aims at learning a structure restorer. It is composed of alternating axial [34] and standard attention blocks to balance performance and effi-ciency. Note that our TSR can achieve much better structure recovery compared with CNNs [13], [21]. Given restored L-Edges, E-NMS is leveraged to eliminate ambiguous blur near the boundaries. Then the output grayscale priors are upsampled by SSU which is consisted of 4-layer CNN to meet the desired resolution. Then, the gated convolutions [35] based SFE encodes prior features and transfers them to a pre-trained FTR with ZeroRA. MPE is proposed to express both distances and directions from unmasked regions to masked ones.
Remarkably, though the FTR is derived from LaMa, we still propose several novel techniques to improve the stable training and thus significantly boost the performance of FTR. Specifically, we promote the maxpool as the mask resizing strategy of PatchGAN instead of the nearest in LaMa. It enables stable training as empirically evaluated in Sec. 3.3. Furthermore, compared with the vanilla attention mechanism, LKA implemented by integrated convolutions [32] is further repurposed in FTR, and enjoys both large receptive fields and scale invariance. Thus LKA greatly strengthens the FTR in HR inpainting. To our knowledge, this is the first time to use extremely large kernels in inpainting tasks.
Finally, we conduct extensive experiments on several datasets, including Places2 [36], ShanghaiTech [29], NYUDepthV2 [37], MatterPort3D [38] and our newly collected HR-Flickr, reveal that our proposed model outperforms other state-of-the-art competitors. The HR-Flickr dataset contains a group of high-quality images, and thus serves as the testbed for the HR image inpainting.
We highlight several contributions in our ZITS++: (1) The TSR module is proposed to learn grayscale structural priors. The attention-based TSR can efficiently learn superior holistic structures with long-range dependency. (2) We study various priors including canny edges [39], wireframes [29], L-Edges [40], gradients, semantic segmentation, RTV [41], LR-RGB and Histograms of Oriented Gradients (HOG) [42]. Generally, L-Edges, wireframe lines and gradients work better than others, as empirically discussed and compared in Sec. 3.6 and Sec. 4.4.1 respectively. Thus they are introduced as structural priors to ZITS++. As compared in Fig. 2, canny edges used in our conference version [1] extract many trivial edges from textures, while edges from L-Edges (CATS) are much cleaner on the texture regions. The L-Edges give meaningful holistic structures, and can be well recovered by TSR to improve the inpainting as in   [39] and (c) CATS [43] learning based edge detection. We use σ = 2 for canny, which is also adapted by most other inpainting methods [12], [13], [21]. Note that CATS edges have much clearer boundary structures and ignore highfrequency textures. But canny produces many meaningless edges from textures in column 1(b).

Fig. 1(a)-(e). (3)
We introduce the E-NMS technique and SSU module to flexibly upsample L-Edges and lines to arbitrary resolutions. This provides critical structural clues for HR inpainting. (4) We present a novel ZeroRA strategy to incrementally incorporate the structural priors into a pretrained FTR. This can largely save computation compared with training from the scratch. Critically, we also propose a novel masking positional encoding is proposed to improve the generalization of the HR inpainting. (5) Our FTR is further improved with the maxpool mask resizing strategy and LKA for a more stable training and better performance. (6) Additionally, we contribute to the community a highquality HR image dataset -HR-Flickr, as the testset for image inpainting. Compared with our conference version [1], we explore more structural priors for inpainting and further improve the module designs of TSR and FTR. We also provide comprehensive discussions about priors for inpainting. We contribute a new HR-Flickr dataset, and conduct extensive inpainting experiments on various resolutions.

Image Inpainting
Many classical approaches are explored to solve inpainting problems, which can be grouped as diffuision-based [44] and patch-based [45] methods. Unfortunately, these methods are typically restricted to low-level image features, and hard to generate expressive results with large holes. Learning based methods equipped with GANs achieve great success in image inpainting [21], [46], [47], [48]. Concretely, Pathak et al. [46] show that the adversarial training is critical to generate clear results for the inpainting. Furthermore, many bespoke inpainting convolutions are designed [35], [49], [50] to gracefully model generalized features for masked regions. Moreover, the attention mechanism is also well explored in the inpainting [15], [18], [35], which takes long-range relation and aggregates reliable features from unmasked regions to masked ones. Recently, StyleGAN [17] based co-modulated inpainting models [47], [48] have achieved high-quality generations enhanced with the style learning. Nevertheless, they [47], [48] are quite likely to hallucinate incompatible image content over the mask. Although these pioneering works have tackled various problems in inpainting, they are generally incapable of reconstructing structural priors to maintain the faithful inpainting results. Transformers for Image Inpainting. Transformers [51] achieved good performance on many vision tasks by learning long-range interactions on sequential image patches. Transformer is also employed to inpaint image in [14], [20] at low resolution, and further guides the GAN-based CNN to produce high-quality results. Unfortunately, directly using transformers to learn large image patches demands huge memory footprint and computations. In contrast, this work utilizes the transformer to build the LR holistic structure reconstruction, which helps to guide the HR image inpainting. Note that the upsampling for grayscale structures enjoys less ambiguities and information loss.

Inpainting by Auxiliaries
Auxiliary information such as canny edges [21], [27], segmentation maps [22], [23], L-Edges [26] and gradients [27] are shown very useful to inpainting. Specifically, EdgeConnect [21] utilizes canny edges to help inpainting images with certain structures. E2I [26] firstly tried to restore images with a two-stage inpainting model enhanced by HED [40]. Guo et al. [12] propose a two-stream network for image inpainting, which models the structure constrained texture synthesis and texture-guided structure reconstruction in a coupled manner. SGE-Net [22] iteratively updates the semantic segmentation maps and the corrupted image. Moreover, RTV is used to filter high-frequency textures for the structural recovery [24] and the decoupling structure/texture feature learning [25]. Our previous work [13] further proposes learning a sketch tensor space, composed of canny edges and lines for inpainting man-made scenes. In this paper, inspired by the pilot study on auxiliary priors in Sec. 3.6, L-Edges and lines are employed to model the structural priors. However, quite differs from [13], the transformer is explored to model structural priors in ZITS++. Particularly, preliminary investigations [52] have shown the excellent capability of transformer in modeling structural relationships for natural image synthesis.

CNNs with Large Kernels
CNNs with large kernels attract some research attention recently. Except for the global CNN learning in the Fourier domain [53], Ding et al. [54] propose to enlarge the kernel size to 31×31 with several tricks and achieve better performance in many downstream tasks. The kernel size is further enlarged to 51 with the sparsity in [55]. To balance the computation, large kernel convolutions are decomposed into a depth-wise convolution, a depth-wise dilation convolution, and a point-wise convolution in [32]. We show that such approximately large kernels can work properly in the image inpainting compared with vanilla convolutions, dilated ones, and even FFC.

METHOD
Overview. The whole pipeline of ZITS++ is illustrated in

Transformer Structure Restoration
Since the transformer shows an ability to get expressive global structure recoveries [14], we leverage the capacity of the transformer to restore holistic structures in a relatively low resolution. For the input masked priors P, and mask M in 256 × 256, we firstly downsample them with three convolutions to reduce computation for attention learning. Such simple convolutions can also inject beneficial convolutional inductive bias for vision transformers compared with the patch based MLP embedding [56]. Then we add a learnable absolute position embedding to the feature at each spatial position and get X ∈ R h×w×c for the input to attention layers, where both the height h and width w are 32, while c = 256 is the feature channel.
To overcome the quadratic complexity of standard selfattention [51], we alternately use axial attention modules [34], [57] and standard attention modules as illustrated in Fig. 3(top-left). The axial attention module can be implemented easily by adjusting the tensor shape for row-wise/column-wise and then processing them with dot product-based self-attention respectively. To improve the spatial relation, we also provide Relative Position Encoding (RPE) [58] for each axial-attention module. For the input feature X ∈ R h×w×c , we suppose that x ri,rj , x ci,cj ∈ R c mean feature vectors of rows i, j and columns i, j of X. Then the row and column-wise RPE based axial attention scores A row , A col can be written as where W rq , W rk , W cq , W ck are trainable parameters for query and key in row and column; R row i,j is the trainable RPE value between row i and j, and R col i,j means the RPE value between columns i, j. Then, the attention scores are processed by the softmax operation. To stabilize the training, we advocate the pre-norm trick in [59]. Compared with the O(n 2 ) complexity of the standard self-attention, the axial attention only has O(2n 3 2 ), which allows us can handle more attention layers for a better model capacity. Additionally, we also remain some vanilla attention modules in TSR for learning better global correlation.
After the encoding of stacked transformer blocks, features are upsampled by three transpose convolutions for outputting structuresP in 256×256 resolution. We utilize various priors; and loss functions of these priors are specified in Sec. 3.6. Note that the motivation of processing the structural priors in 256×256 is to save the computation and memory for the attention-based TSR. And we further show our SSU module in Sec. 3.2 can flexibly upsample these priors to higher resolution.

Upsampling Priors
Upsampling Lines. To capture holistic structures for higher resolution images, we should upsample the generated priors P to arbitrary scales without obvious degeneration. Particularly, upsampled structures can achieve superior performance as empirically analyzed in Sec. 4.4.1. However, vanilla interpolation-based resizing causes the aliasing effect to lines as well as canny edges as in Fig. 4(f)-(i). Such artifacts are more serious for large image sizes, and deteriorate the inpainted results. To address this issue, we train   a CNN-based SSU to upsample lines to a doubled size. Since lines obtained from a wireframe parser have good discrete representations [29], [60], i.e., a line can be indicated as positions of two endpoints and their relation, we can draw line maps in various resolutions without any ambiguities as in Fig. 4(d) and Fig. 4(e). Through the iterative calling, we can get high-quality lines at higher resolutions. Remarkably, despite lines can be upsampled successfully, canny edges fail to get correct results with the same training strategy as shown in Fig. 4(j). Because there are ambiguities in the canny edge from different image sizes as in Fig. 4(b) and Fig. 4(c). Interestingly, we find that if the upsampling module is trained in lines, it can also produce smoothed canny edge maps at higher resolution, which thanks to the generalization of the network as in Fig. 4(k). Such a good property can also be generalized for upsampling L-Edges. Upsampling L-Edges. We promote the learning-based edges -CATS [43], rather than canny edges in [1]. Such priors can also be well generalized to the cases of higher resolution with the newly introduced E-NMS and our SSU trained with lines. As in Fig. 5, we take E-NMS in edge evaluation of [61] and implement E-NMS in PyTorch to filter uncertain edge predictions.  Figure 6. The influence of different mask resizing strategies for LaMa on the Places2 subset of 25,000 training and 500 validation images over 5 scenes. LaMa worked with nearest resizing is unstable. Table 1 Quantitative ablations of FTR based on the Places2 subset of 25,000 training and 500 validation images over 5 scenes. 'AddConv' means additional convolution types utilized in the upsampling and downsampling modules of FTR. 'Conv2D+' indicates using more 2D convolutions to balance the computation compared with FFC and LKA.
Numbers in brackets beside the LKA mean effective kernel sizes. are discussed in the supplementary.

Fourier CNN Texture Restoration with LKA (FTR)
For the texture restoration, we adopt Fourier convolutions [53] from LaMa [2] for learning in the frequency domain. Note that FTR also works as the pre-trained inpainting model for our ZeroRA finetuning with structural priors from TSR. As in Fig. 3, FTR is an autoencoder based generator. The key module of FTR is the FFC layer, which is consisted of two branches: 1) the local branch uses conventional convolutions and 2) the global branch convolutes features after the fast Fourier transform. Then two branches are combined for larger receptive fields and local invariance during the inpainting [2]. Unfortunately, LaMa didn't explicitly learn reasonable holistic structures, which is addressed in our ZITS++. Improving FTR from LaMa. Although LaMa can produce high-quality inpainted results, the training of LaMa is quite unstable even with the EMA as compared in Fig. 6(a) on the Places2 subset. We expose that such an unstable training suffers from the resizing way of PatchGAN [62]. For the patch-wise fake loss optimization of the discriminator, LaMa leverage the nearest resizing, where some partially masked patches are seemed as unmasked ones as in Fig. 6(d). Instead, we find that using maxpool resizing as Fig. 6(c) can greatly stabilize the training process. It means that partially masked patches should be considered as 'fake' patches rather than 'real' ones for inpainting. Besides, we upgrade all activation in the generator of FTR to Swish [63], which has been shown to be effective in many generation works [17], [64]. Detailed ablation studies of the improved FTR are shown in Tab. 1. LKA Inpainting Module. It has been well explored that large receptive fields are important for image inpainting [2], [50]. FFC used in LaMa [2] has the global receptive field in the frequency domain. However, FFC fails to learn reliable feature relations between masked regions and unmasked ones. On the other hand, various attention mechanisms with long-range dependencies are also incorporated into many inpainting methods [15], [18], [48]. But these attention-based methods inevitably overfit certain resolutions without scale invariance. To unify the advantages of both CNN and attention, we propose to inject large-kernel based convolutions to FTR. We follow [32] , and a point-wise convolution (Conv2d 1×1) as in the lower right corner of Fig. 3. LKAs are incorporated into upsampling and downsampling blocks of FTR, which are orthogonal to FFCs. Because LKAs devote to learning better feature representations for masked regions with large receptive fields, FFCs tend to facilitate the global learning for regular textures in the frequency domain. Furthermore, we also use shortcuts with 3×3 convolutions as Feed Forward Networks to ensure the generalization of LKA. Our empirical ablations in Tab. 1 show that LKAs work better than FFCs and vanilla convolutions. Thus LKA with K = 21 is used in ZITS++.

Structure Feature Encoder (SFE)
For the given restored priorsP in arbitrary scales, a full convolutional network is employed to process them into a feature space. Our SFE is also an autoencoder model with 3 layers downsampling convolutions (encoder), 3 layers residual blocks with dilated convolutions [16] (middle), and 3 layers upsampling convolutions (decoder). For the encoder and the decoder in SFE, we use Gated Convolutions (GCs) [35] to transfer useful features selectively. GC learns another sigmoid activation with the same channels. Then the sigmoid features are multiplied to the convoluted ones as outputs. Although GCs are widely used in image inpainting for the better generalization to irregular masks, we use GCs to filter useful features to FTR. Because the grayscale sketch space is sparse, and not all features are necessary for the inpainting. Then, 4 coarse-to-fine feature maps {S k } 3 k=0 from the last middle layer and 3 decoder layers are selected to transfer structural features to FTR as where M indicates the resized binary mask.

Masking Positional Encoding (MPE)
Although the zero-padding in CNNs can provide some position information [65], it only contains information about spatial anchors [30]. Therefore, central generated regions from GANs tend to repeat meaningless artifacts without specific position encoding. When the image size is large, the effect of zero-padding will be further weakened, which causes more repeated artifacts [31] to generators.
During the inpainting, position information for unmasked regions is unnecessary, because the model always knows ground truth unmasked image regions. However, we think that position information is still critical for masked regions, especially when mask areas are large for HR images. Limited by the receptive fields of CNNs, the model with large masks may lose the direction and position information, which causes meaningless artifacts. Although FFC can extend the feature learning to the frequency domain, it is insensitive to distinguish masked or unmasked regions. Therefore, we propose to use position encoding in masked regions called MPE for the image inpainting, which is orthogonal to and improves upon the FFC in FTR. Specifically, our MPE can be expressed as the masking distance E dis and the masking directions E dir as shown in Fig. 7. Given an inversed 256×256 binary mask, where one indicates unmasked regions and zero indicates masked regions, we use a 3×3 all-one kernel to calculate the masking distance D dis for each position in masked regions as shown in Fig. 7(b). Then, the distance is clipped and mapped by the Sinusoidal Positional Encoding (SPE) [51] to get where i indicates the channel index; D max = 128, and d = 64 means the total channels of E dis , which is the same as the first convolution of FTR. Since SPE can only provide absolute positional information [30], E dis can be further resized by the nearest interpolation to various scales during the training for learning relative positional information in arbitrary resolutions. For masking directions, we use 4 different binary kernels to get the 4-channel one-hot vector D dir ∈ R 256×256×4 . Values of D dir depend on which kernel covers the masked regions firstly. D dir shows the nearest direction from a masked position to an unmasked one as shown in Fig. 7(c). Note that the masking direction is a multi-label vector, because a pixel may have more than one shortest direction. Then D dir is projected to a d dimension features with learnable embedding parameters W dir ∈ R 4×d as E dis and E dir are added as MPE to the first layer of FTR.

Zero-initialized Residual Addition (ZeroRA)
Since most inpainting methods are based on sophisticated GANs nowadays, training the inpainting model incrementally is non-trivial. However, benefiting from various auxiliary information [13], [21], [22], incrementally training is flexible to improve the image inpainting. To improve the pre-trained inpainting model (i.e., FTR in ZITS++) incrementally with holistic structures, we propose to use ZeroRA, which has been leveraged in [33] to replace the layer normalization in the transformer. Compared to training from scratch with auxiliary information, ZeroRA can save 11 days' computations on 256 × 256 resolution Places2 with 3 V100 16 GB GPUs. The idea of ZeroRA is simple. For the given input feature x, the output feature x ′ is got from adding a skip connection with function F to x with a zeroinitialized trainable residual weight α as For simple linear-based models, if α is initialized in zero, the input-output Jacobian will be initialized to 1, which makes the training stable. For more complex cases, experiments in [33] also prove the effectiveness of ZeroRA. Since ZeroRA can replace the layer normalization in the transformer, it can also improve the expressive power of the model without degrading variances to early layers.
In our case, we use ZeroRA to incrementally add structural information from SFE to FTR. Specifically, 4 zeroinitialized {α k } 3 k=0 are utilized to fuse 4 related feature maps {S k } 3 k=0 from SFE. For the feature X k of FTR encoder layer k, which is based on Conv-BatchNorm-ReLU, we add residuals as follows There is another advantage of the ZeroRA based incremental learning. The model output is equivalent to the pretrained one at the beginning of finetuning, which can effectively stable the training, and transfer necessary information adaptively. Our ablation studies show that the ZeroRA is important to incrementally finetune the pre-trained inpainting model with additional information.

Loss Functions
We adopt the same loss functions as [2], which include L1 loss, adversarial loss, feature match loss, and High Receptive Field (HRF) perceptual loss [2]. Firstly, L1 loss is only calculated within the unmasked regions as where M indicates 0-1 mask that 1 means masked regions; ⊙ means the element-wise multiplication;Î,Ĩ indicate the ground truth and predicted images respectively. The adversarial loss is consisted of the discriminator loss L D and the generator loss L G . Moreover, we only regard features from masked regions as fake samples in L D . The PatchGAN [62] based discriminator is written as D and the combination of FTR and SFE can be seen as the generator G, Then the adversarial loss can be indicated as where L GP = EÎ||▽ÎD(Î)|| 2 is the gradient penalty [66] and λ GP = 1e − 3. We also use the feature match loss [67] L f m , which is based on L1 loss between discriminator features of true and fake samples. L f m is usually used to stable the GAN training. It can also slightly improve the performance. Furthermore, we use the HRF loss L hrf in [2] as where ϕ hrf indicates a pre-trained segmentation ResNet50 with dilated convolutions. As discussed in [2], using HRF loss instead of the perceptual loss can improve the quality of the inpainting model. The final loss of our model in the incremental training can be written as where λ L1 = 10, λ adv = 10, λ f m = 100, λ hrf = 30.

Various Priors for Inpainting
In this section, we introduce various priors, which are roughly categorized as classical priors, and learning-based priors. The classical priors have canny, gradients, RTV, HOG, and LR-RGB, while latter one will consider the wireframes, L-Edges, and semantic segmentation. Learningbased priors employs the deep models to extract priors from images. These priors will be negatively affected by masked images [22], as no visual content is available. Visualizations of these priors are in Fig. 8. In this subsection, we still use the Place2 subset of 25,000 training and 500 validation images over 5 scenes for the pilot study.

Classical Priors
Canny Edges. Following [21], we set σ = 2 as Gaussian blur coefficient for the 256 × 256 canny detection. Canny edge priorsP ce are optimized with Binary Cross-Entropy (BCE) loss in TSR. We use the mask-predict [68] trick and sigmoid temperature parameters for better canny edge predictions, detailed in our supplementary. Gradients. The gradient prior is also used as an auxiliary loss for inpainting in [27]. As in [27], we use Sobel filters to extract ground truth RGB gradientsP g in two directions, i.e., horizontal and vertical gradient maps. Gradient priors P g are optimized by L 1 loss with edge regularization as [27]: where * and ⊙ means convolution and element-wise multiplication; g is a 10 × 10 Gaussian filter with standard deviation 1; C is canny edge; β 1 = 0.1; β 2 = 20. Relative Total Variation (RTV). Xu et al. [41] propose to use RTV map to decouple structure and textures. Such decoupling has been demonstrated useful to image inpainting [24], [25]. We follow [24] to use Gaussian kernel σ = 3 and smooth degree λ = 0.015 to extract structural features from 256 × 256 images. Then, L 1 loss is taken to optimize RTV priorsP rtv .
Others. Additionally, we also compare the priors of HOG [42] and Low-Resolution RGB Pixels (LR-RGB), as details are in the supplementary.

Learning-based Priors
Lines. The common solution of wireframe parsing is to achieve endpoint heatmaps at first, and then verify valid endpoint pairs through verification modules [29], [60], [69]. However, these wireframe parsers are unable to produce reliable endpoint heatmaps from corrupted images as discussed in [13]. Thus we employ the masking augmented wireframe parser -LSM-HAWP [13] to extract lines from masked images. Specifically, we use anti-aliased lines to link endpoint pairs to achieve the ground truth line priorŝ P l . For simplicity, we directly mask ground truth priorŝ P l as masked ones P l for training. During the inference, LSM-HAWP can be well generalized to masked images as illustrated in Fig. 9. The reconstructed linesP l are also in grayscale, so we can optimize them with BCE. L-Edges. L-Edges such as HED [40], CATS [43], and Dex-iNed [70] enjoy good property to decouple structures and textures. So it is interesting to understand which type of L-Edges is in good favor of the inpainting task. As in Fig. 8(h)-(j), we give some observations: HED edges are more indecisive near boundaries; CATS edges can get clearer results, but lose some continuous information; DexiNed edges can extract very dense L-Edges greatly benefited by learning from fine-grained annotations. However, these L-Edges also suffer from corrupted images, as the example of CATS edges in Fig. 9(d). To this end, we make TSR work for both edge detection and recovery trained with BCE on masked regions. This strategy can be well generalized to masked images without information leakage as in Fig. 9(e). And, only one forward pass is needed to achieve binary L-Edge priorsP le from TSR. Semantic Segmentation. Priors from semantic segmentations show good performance in inpainting [22], [23]. But such priors are largely influenced by corrupted images [22].
Here we leverage an off-the-shelf ADE20K [71] pre-trained segmentation SETR model [72] to extract ground truth segmentation priorsP s from unmasked images.

Discussion of Various Priors
In this work, we utilize the L-Edge CATS, wireframe lines, and gradients as our priors for the inpainting on the 256×256 images. Gradients are not used for the inpainting at higher resolution, as we empirically compare and discuss various priors in Sec. 4.4.1.

Why L-Edges outperform other priors?
In general, L-Edges produce more significant boundaries rather than trivial edges from texture (canny), as visualized in Fig. 2. Such structural boundaries of L-Edges reflect the scene-level understanding to the image. Here we adopt CATS from all L-Edges, since CATS outperforms HED and DexiNed for inpainting. Particularly, HED and DexiNed give blurry edges over the image regions of boundaries and high-frequency details respectively as illustrated in Fig. 17. Furthermore, CATS can be well generalized to HR cases with the E-NMS and upsampling of SSU in Sec. 3.2. In contrast to other priors, such as gradients and RTV, SSU cannot tackle the upsampling of them, which leads to degradation and artifacts for unseen scales (1024) during the training as in Tab. 8 and Fig. 16. For the consistent LR inpainting (256×256), our model further includes gradient that outperforms all other priors.
Effect of Wireframe Lines. From Tab. 7, the wireframe line works compatibly with CATS and gradients, and prominently improve FID and LPIPS for both image inpainting at 256 × 256 and 512 × 512 resolution. Such phenomenon indicates that lines can provide simple but strong structural priors to overcome artifacts during the generation as in Fig. 17(b), especially for large masking regions and HR inpainting. Moreover, lines enjoy great advantages for manmade scenes as studied in [13]. In practice, by using the same TSR, lines can be jointly trained and inference with CATS at negligible additional cost.  [74] to test the HR inpainting in HR-Flickr, which include thin, medium, and thick brush masks. The HR-Flickr dataset will be released to the community.

Implementation Details
Training Settings. Our ZITS++ is implemented with Py-Torch. For the training of TSR, we use the Adam optimizer of learning rate 6e-4 with 1,000 steps warmup and cosine decay. TSR is trained with 150k and 400k steps for Indoor and Places2. On the other hand, we first train the FTR with Adam optimizer of learning rates 1e-3 and 1e-4 for generator and discriminator respectively. And FTR is trained with 100k steps on Indoor and 800k steps on Places2. Then, we  .15 0.917 14.61 0.090 25.13 0.884 1.19 0.094 incrementally finetune them with ZeroRA for 50k and 150k steps on Indoor and Places2 respectively, and reduce the generator learning rate to 3e-4. Besides, we warmup the learning rate for training the SFE with 2,000 steps. For the training of TSR and FTR, input images are resized into 256×256. For the incremental finetuning, we separately train two versions of ZITS++, which are the version trained in 256×256 and the version trained in random size from 256 to 512. The second model can handle some situations with higher resolution inputs. And the MPE is also changed to a relative position encoding for the random size training. Mask Settings. To tackle the real-world object removal task, we follow the mask setting from [13], which includes irregular masking brushes and segmentation masks with masking rates from 10% to 50%. Different from [13], we additionally use some LaMa irregular masks [2] (both thin and thick types) for a better diversity, and randomly combine irregular and segmentation masks with 20% to improve the learning difficulty. Competitors. Our conference version [1] uses canny edges and lines is denoted as ZITS [1], while ZITS++ indicates our improved version in this paper enhanced by the priors of CATS, gradients and lines, as well as several novel modules and techniques. We compare the proposed model with other state-of-the-art methods, which include Edge Connect (EC) [21], Contextual Residual Aggregation (Hi-Fill) [15], Multi-scale Sketch Tensor inpainting (MST) [13], Co-Modulation GAN (Co-Mod) [47], Mask-Aware Transformer (MAT) [48], and Large Mask inpainting (LaMa) [2]. All competitors are compared in the Places2. We also retrain EC, MST, and LaMa for the Indoor dataset to discuss the structure recovery. Note that the LaMa compared below are all trained with the same total steps as ZITS++.

Quantitative Comparisons
Quantitative Inpainting Results. In Tab. 12, we utilize PSNR, SSIM [75], FID [76], and LPIPS [77] to assess the performance of all compared methods including our conference version on the Indoor and Places2 datasets in 256×256 with mixed segmentation and irregular masks. More results with different masking rates are shown in the supplementary. For Indoor, our ZITS++ and ZITS achieve the best and the second best results on all metrics. MST can get slightly better results compared with EC, which is benefited by the usage of lines. LaMa can get more acceptable FID and LPIPS while our ZITS can achieve significant improvements based on LaMa due to the seamlessly embedded structural information and positional encoding. Note that the gap between Table 3 Quantitative Precision (P.), Recall (R.) and F1-score (F1) of edges and lines on Indoor and Places2.

Qualitative Comparisons
Qualitative Inpainting Results. We show the qualitative inpainting results of Indoor in Fig. 10 and Places2 in Fig. 11. Compared with other methods, our ZITS++ can tackle more reasonable structures, especially our method can obtain clearer borderlines. Furthermore, ZITS++ achieves prominent improvements in the structure recovery compared with LaMa; and outperforms the state-of-the-art MAT [48] with much more faithful results. Note that both LaMa and ZITS++ are trained with the same steps. Qualitative Results of Edges and Lines. We compared the structure recovery results in Indoor of Fig. 12, which are compared between our transformer-based TSR and CNNbased model from MST. TSR can achieve more reasonable and expressive results of both edges and lines. More qualitative structural results are shown in the supplementary.

Ablation Studies
Quantitative ablation studies on Indoor are shown in Tab. 5. MPE and GCs can slightly improve the performance of FTR. Besides, if adding structural information from TSR without ZeroRA, the improvement is limited. So ZeroRA is useful for incremental learning with a good convergence. Moreover, the full model achieves the best performance. Effects of MPE. We further exploit the effects of MPE in HR inpainting. FTR is trained without MPE at first. Then we use the ZeroRA technique to finetune the model with and without MPE of the same steps. Results in Tab. 4 show that the simple MPE based finetuning effectively improves the 512-inpainting in FID. From Fig. 13, ZITS with MPE generates images with natural and smooth colors.

Comparisons between Canny and CATS Edges.
Since the edge prior is simultaneously utilized in our conference model ZITS [1] (canny) and the newly proposed ZITS++ (CATS), we further compare their performance in Fig. 14 and Fig. 15. From Fig. 14, CATS can achieve more reasonable structure recovery benefited from the superior scene understanding from L-Edge. Moreover, CATS results pay more attention to valuable object boundaries rather than textural canny edges filtered by gradients. Such a decoupling benefits both TSR and FTR for their own functions as mentioned in Sec. 3.6.3. Besides, from the HR inpainting results of 1024×1024 shown in Fig. 15, our ZITS++ address a failed HR case in ZITS with better generation of distant buildings. Therefore, the CATS enhanced ZITS++ enjoys superior performance for image inpainting.  Figure 11. (A) Places2 256×256 results compared with EC [21], HiFill [15], MST [13], Co-Mod [47], LaMa [2], and our ZITS++. (B) Places2 512×512 results compared with MAT [48], LaMa [2], and our ZITS++.        Fig. 17. Note that the baseline model is also finetuned with the same steps on the subset. All priors can improve the performance based on the baseline 1 . Both lines and edges after E-NMS can be upsampled by our SSU, while other priors suffer from artifacts in boundaries during the HR inpainting as in Tab. 8 and Fig. 16. HOG and LR-RGB have discouraged improvements in Tab. 6 without proper structures. Our conference version (Canny+Line) gets moderate performance compared with other priors, while L-Edges (HED, CATS, DexiNed) achieve better results in FID. Furthermore, CATS enjoys best performance among three L-Edge priors. Moreover, semantic segmentation works well, but we have only reported the upper bound of semantic performance here due to the gap of segmentation results 1. Wireframe lines do not work as an independent prior for image inpainting in this paper. Because lines are more effective in man-made scenes, while some cases have no lines at all. The effectiveness of line based on canny edge has been demonstrated in MST [13]. Table 9 Quantitative results of 512×512 in Indoor and Places2, and 1024×1024 MatterPort3D. The best results are in bold, while the second best ones are underlined. from corrupted images. Gradients also greatly facilitate the inpainting performance, especially in PSNR and SSIM. Though RTV is also competitive, but it still enjoys less priority compared with L-Edges and gradients. Besides, gradient and RTV fail to be generalized to HR with the SSU.

PSNR↑ SSIM↑ FID↓ LPIPS↓
Prior Combinations. We further try to compare prior combinations among CATS, gradient, and line in Tab. 7. Structural priors achieve prominent improvement to the baseline. Specifically, CATS and gradient can work compatibly and achieve superior results of all metrics. Besides, wireframe lines benefit the both 256 × 256 and 512 × 512 inpainting with better FID and LPIPS. Note that the gradient can not be generalized to untrained scales. So we only use it for the 256×256 inpainting.

Results of High-Resolution Inpainting
We also compare the results of HiFill, Co-Mod, LaMa, MAT, our ZITS and ZITS++ in Places2(512) of Tab. 9. Besides, LaMa, ZITS, and ZITS++ are further compared in Indoor(512) and MatterPort3D(1k) in Tab. 9. Comparisons of HR-Flickr are shown in Tab. 10. ZITS and ZITS++ are firstly trained in 256×256 and then finetuned with dynamic resolutions from 256 to 512 with 150k steps. Models tested in Indoor(512) and MatterPort3D(1k) are both trained in Indoor training set, while methods tested for HR-Flickr are trained on Places2. For the Places2(512), we randomly select 1,000 samples from 36,500 for the 512 testing. Our ZITS++ can outperform LaMa and even previous ZITS, which illustrates the effectiveness of the CATS edges and the updated FTR. Besides, ZITS++ can also get better 1k results in MatterPort3D. More HR results can be seen in the supplementary.

CONCLUSIONS
In this paper, we propose an incrementally structure enhanced inpainting model called ZITS. We use a transformerbased structure restorer to get much better holistic structures compared with previous methods. Then, a novel ZeroRA strategy is leveraged to incorporate auxiliary structures into a pre-trained inpainting model with a few finetuning steps. Table 10 Quantitative results of HR-Flickr with thin, medium, and thick types of masks defined in [74]. The best results are in bold.

HR-Flickr(1K) HR-Flickr(2K) HR-Flickr(4K)
Mask  The proposed masking positional encoding can further improve the inpainting performance. Moreover, we further study different image priors for inpainting, and select to leverage CATS edges [43] instead of canny [39] used in the conference version [1], which provides more informative structures to the inpainting model. Besides, we comprehensively upgraded LaMa [2] that used as the pre-trained FTR of ZITS with LKA and several useful techniques. The newly proposed ZITS++ can achieve significant improvements based on the state-of-the-art model in experiments of various resolutions, even outperform our previous ZITS.

BROADER IMPACTS
All generated results of both the main paper and the supplementary are based on learned statistics of the training dataset. Therefore, the results only reflect biases in those data without our subjective opinion. This work is only researched for the algorithmic discussion, and related societal impacts should not be ignored by users.

INTRODUCTION OF OTHER PRIORS
HOG [42] is a feature descriptor of gradient orientation distributions within local patches. MaskFeat [78] finds that HOG-based self-supervised pre-training video model can achieve very good performance in ImageNet. Therefore, HOG has been studied as a prior to inpainting. As to [78], we utilize a vanilla ViT [79] to inpaint HOG with masked inputs in RGB channels. Each cell comprises 8 × 8 pixels with 9 orientations. Thus a 256 × 256 image results in P h ∈ R 32×32×27 HOG features. L 2 loss is used to minimize the distance between the predicted HOG featuresP h and the original onesP h . Low-Resolution RGB Pixels (LR-RGB). RGB pixels are the most intuitive priors for inpainting. To address quadratically costly computation of transformer, ICT [14] leverages LR-RGB priorsP rgb (32 × 32 in our implementation) to guide the inpainting. To further reduce the color space, all RGB combinations are clustered to 512 discrete K-Means centers on the whole ImageNet as [52]. Moreover, masked pixels are replaced with trainable tokens [MASK]. Then generatedP rgb are bilinear upsampled to the stage-2 model. Thus stage-2 can be also seen as a super-resolution task, while stage-1 is working as LR inpainting. Since the color space is clustered to 512 discrete centers, the optimization can be seen as a classification. Therefore, the objective of P rgb is to minimize the negative log-likelihood with crossentropy (CE) for masked tokens.

DEATILS OF E-NMS FUSION
Since the original CATS edge can be seen as a continuous probability image, which is blurred near the boundary; and the bilinear upsampling will further aggravate the blur, which eventually degrades the performance of image inpainting, as shown in Fig. 18(b)(e). So we use E-NMS to work as a non-maximum suppression of the edge to preserve the sharp edge as in Fig. 18(c). On the other hand, we find that E-NMS loses a lot of low-probability regions, which may contain informative clues for inpainting models. Therefore, we decide to make a trade-off between the continuous CATS and the E-NMS CATS. We keep areas as the original CATS with probability less than 0.25, while areas with probability larger than 0.25 are replaced with CATS after E-NMS. The fused edge map and the related inpainting result is shown in Fig. 18(d) and Fig. 18(f) respectively.

DETAILED NETWORK SETTINGS
We show some detailed network settings in Tab. 11. Besides, the transformer block and Fast Fourier Convolution (FFC) block [2] have been introduced in the main paper. The dilated resnet block is from the middle layer of [21] with dilate=2.

KANIZSA TRIANGLE INPAINTING
We also conduct an interesting experiment on completing Kanizsa Triangle, which is utilized as the teaser figure of David Marr's book [80]. As contours in an outline drawing can convey meaning and play a crucial role in human vision, understanding and inpainting ideographic images are systematically tied to people's visual perception and cognitive processes. For instance, in Fig. 19(a), we can observe partially occluded disks and another triangle under a nonexistent white triangle in the Kanizsa triangle [81]. Benefited from transformer's better understand of global structure than CNN [14], our ZITS++ can inpaint more meaningful and complete structure than CNN-based EC [21] as in the first row of Fig. 19. And our final inpainting results exchange the top/down positions of the black-sided triangle and the nonexistent white one in the Kanizsa triangle.

UPSAMPLING ITERATIVELY WITH SSU
Our Simple Structure Upsampler (SSU) introduced in Sec 3.2 can also work iteratively for larger image sizes. First, we should process the output edges I ′ e and lines I ′ l of SSU through shifted sigmoid as where γ = 2, β = 2 in our evaluation, and γ, β are randomly selected from [1.5, 3] for the finetuning. Since the output size of SSU is doubled, we can repeat the inputs I e , I l ∈ R h×w for q times to achieve I ′ e , I ′ l ∈ R 2 q h×2 q w . Then, the outputs can further be resized with the bilinear interpolation for the target size. In general, our SSU can get good and robust upsampled results for large sizes as shown in Fig. 20.

OUR HR-FLICKR DATASET
For the HR cases with more diversity, we newly release a group of high-quality images collected from Flickr with downloading permissions from owners, called HR-Flickr. HR-Flickr consists of 500/500/490 HR images with resolutions 1k, 2k and 4k respectively as shown in Fig. 21. Images from each group of specific resolution have maximum side lengths which are equal to the corresponding group resolution (1024, 2048, 4096). Moreover, images with persons are eliminated by segmentation methods and manual checking from HR-Flickr to reduce the privacy risk as far as possible. Compared with other HR dataset, such as DIV2K and Flickr2K [73], our dataset contains 4k images. Besides, HR-Flickr has more diverse and challenging cases for image inpainting, while we manually filter images with too simple textures or monotonous scenes.

SUPPLEMENTARY EXPERIMENTS
In this section, we provide some more qualitative and quantitative results to show the effects of our proposed components. Moreover, some details about the post-processing are also discussed.

More Qualitative Results
More qualitative 256×256 results of Indoor and Places2 are shown in Fig. 22 and Fig. 23. Qualitative 512×512 Places2 results are shown in Fig. 24. Note that our method not only achieves better results in many man-made scenes, but also gets competitive results in natural scenes benefited from MPE and edges.

Quantitative Results with Different Masks
We show more quantitative results with different masking rates from 10% to 50% and mixture of segmentation and irregular masks in Tab. 12.

More Structural Experiments
TSR Ablations. For the Indoor dataset, we conducted several ablation experiments on our Transformer Structure Restoration (TSR), and the results are displayed in Tab. 13 and Tab. 14. As illustrated in Tab. 13 and the first two rows of Tab. 14, replacing one standard self-attention module [51] with an axial attention module [34] in our Transformer Block can greatly reduce the GPU memory usage and speed up the model inference while keeping all metrics basically unchanged. Furthermore, we add the relative position encoding (RPE) [58] into our axial attention module, which can boost our results. Note that the RPE must be incorporated with the axial attention module in row-wise and columnwise, while standard attention based RPE costs much more

Effects of Mask-Predict
During the inference, Mask-Predict [68], [82], [83] is used in TSR which is a non-autoregressive sampling method. Mask-Predict predicts all target pixels at the first iteration.
Then we re-mask and re-predict pixels with low-confidence iteratively. Mask-Predict can greatly enrich the generated results without heavy costs. Since our TSR can output a 256 × 256 probability map for edge and line respectively; of course, we can directly use this probability map as the repair result of our edge and line. However, we find that the recall is still insufficient. Fig. 26(b)(c) show that it can only predict a few regions with high confidence. But the probability confidence for the inner masked region is low, which leads to incomplete structures. As a result, we employ the Mask-Predict [68], [82], [83] strategy. It predicts all target pixels at the first iteration. Then we re-mask and re-predict pixels with low-confidence iteratively for a constant number of iterations. Note that we just re-mask edge and line without re-masking the input masked image. This technique can achieve more complete structural information as illustrated in Fig. 26(d)(e). We set Table 14 Ablation studies of TSR on the Indoor dataset, where P., R., F1 mean Precision, Recall, and F1-score.  Figure 21. High-quality images from our HR-Flickr with 1k, 2k, and 4k resolutions.
the total number of iterations to 5 in our experiment for a trade-off between inference time and recall. Fig. 27 shows our outcomes with different Mask-Predict iterations. Note that we ignore pixels with low confidence for each iteration in Fig. 27, so the iteration 1 results of Fig. 27(c) look different from the results without Mask-Predict. The inside portion of the mask can be gradually restored with larger iterations as shown in Fig. 27.

Effects of ZeroRA
We also show line charts of PSNR and FID during the finetuning in Fig. 28 with and without ZeroRA. The blue curve without ZeroRA is unstable at the beginning of the finetuning, while the red one with ZeroRA enjoys better convergence and stability. Because adding extra structural features without ZeroRA leads to dramatic output changing, which harms the vulnerable GAN training.

User Study
We conduct user study on several models to validate the effectiveness of our model from the perspective of human. Specifically, we invite 10 volunteers who are not familiar with image inpainting to judge the quality of inpainted images. On Indoor and Places2, four methods are compared,   [21], MST [13] LaMa [2] and ours. Given the masked inputs, we randomly shuffle and combine the results of four methods together. Then, volunteers are required to choose the best one from each group. As shown in Fig. 29, our method outperforms other three competitors on both two datasets. Especially, our method can achieve a great advantage compared with the baseline method i.e., LaMa.

Results of Rectangular Masks
Here we provide some results of 40% center rectangular masks of 1k Places(512) images without any retraining in Tab. 15. Note that Co-Mod [47] is the only one trained with some rectangular masks while other methods have not been trained with similar masks. Moreover, we compare related qualitative results in Fig. 30. And the classical exemplarbased inpainting [9] is also included. Traditional exemplarbased method fails to work properly and is time-consuming. Co-Mod has hallucinated artifacts instead of generating plausible results. And LaMa results are blur with still high PSNR.

Comparisons of Texture Images
We further compare our method with LaMa on 1,880 texture images [84] in Tab. 16 and Fig. 31, which contain strong periodic textures. Since this dataset is very suitable to LaMa [2], our method still has competitive performance.

Results of MatterPort3D
We use the test set of MatterPort3D [38] to evaluate the effectiveness of our method in the high-resolution structure recovery. MatterPort3D images tested in this paper are  Figure 22. Qualitative 256×256 results of Indoor dataset compared among EC [21], MST [13], LaMa [2], and ours. Zoom-in for details. . Qualitative 256×256 results of Places2 dataset compared among EC [21], HiFill [15], MST [13], Co-Mod [47], LaMa [2], and ours. Zoom-in for details. consisted of 1,965 indoor images in 1280×1024. We resized them into 1024×1024 as shown in Fig. 32. We provide some qualitative results of our method and LaMa compared on MatterPort3D in Fig. 33. For these structural images, our results enjoy better structures.

MORE HIGH RESOLUTION RESULTS
In Fig. 34, Fig. 35, Fig. 37, Fig. 36, and Fig. 38 we provide some object removal instances in large images from 1k to 2k resolutions compared with LaMa [2]. Some cases are selected from the open-source testset of LaMa and others are selected from our HR-Flickr. Note that our method outperforms LaMa in scenes with weak textures such as row 2 in Fig. 34 and row 1 in Fig. 35. For the cases with sparse regular textures and lines (rows 1,3 of Fig. 34), our method can still achieve more clear borderlines. For the cases with dense regular textures (rows 2,3 of Fig. 35), LaMa gets competitive results, which shows that FFC in frequency fields has solved these problems properly. However, our method can also achieve results with less blur that benefited from precise structural constraints. For the larger case with 2048 image size in Fig. 37, our method can still get more consistent result compared with LaMa.    Figure 29. Average scores of Indoor and Places2 for user studies, which are collected from volunteers who select the best one from shuffled inpainted images.