{'fraction_non_alphanumeric': 0.025158868779962096, 'fraction_numerical': 0.008844624475082686, 'mean_word_length': 5.181943487250172, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Restore from Restored: Single-image Inpainting

Recent image inpainting methods show promising results due to the power of deep learning, which can explore external information available from a large training dataset. However, many state-of-the-art inpainting networks are still limited in exploiting internal information available in the given input image at test time. To mitigate this problem, we present a novel and efficient self-supervised fine-tuning algorithm that can adapt the parameters of fully pretrained inpainting networks without using ground-truth clean image in this work. We upgrade the parameters of the pretrained networks by utilizing existing self-similar patches within the given input image without changing network architectures. Qualitative and quantitative experimental results demonstrate the superiority of the proposed algorithm and we achieve state-of-the-art inpainting results on publicly available numerous benchmark datasets.


Introduction
Image inpainting, a post image processing technique that removes unnecessary parts of images, such as subtitles and obstacles, and fills these areas (i.e., missing areas) with new colors and textures, has extensive applications in areas of photo editing, restoration, and even video editing [4,5,1,6,7]. The main goal of inpainting is to create a plausible image that semantically consistent with overall context and should be naturally connected with surrounding region, particularly in the missing area.
Many traditional inpainting approaches based on diffusion technique compute pixel values of the missing area using pixel values of the surrounding area [8,9,10]. Although diffusion-based approaches are effective in recovering small areas (e.g., subtitle removal), they produce blurry results when the missing area is large (e.g., object removal). As an alternative, patch-match-based methods have been proposed to locate similar patches within the given image and copy intensity values of these similar patches into the missing area [11,12,4]. This method can fill the missing area with realistic objects associated with image contents using information that can be obtained from the image. However, (a) (b) (c) Figure 1: Results of our approach. (a) Masked input images. (b) Initially restored results from pretrained inpainting networks in the order of GatedConv [1], EdgeConnect [2], and GMCNN [3]. (c) Results of fine-tuned versions of GatedConv [1], EdgeConnect [2], and GMCNN [3] using the proposed learning algorithm. Restored regions are more natural and consistent with other regions than the initial results.
restoring faces or complex landscapes that require information outside the given image is impossible and inferring the image context sometimes produces irrelevant structures.
The development of deep learning technologies has remarkably improved the performance of image inpainting techniques and made it possible for neural approaches to reconstruct considerably challenging images. First, context encoder pioneered the use of deep neural networks in the image inpainting task and introduced an architecture composed of convolutional encoder and decoder with adversarial loss [13,14]. The encoder computes a latent feature representation from the input image and uses this feature representation to restore pixels in the missing area through the decoder to infer the overall structure in the test stage. This output demonstrates a realistic texture as it approaches the ground-truth distribution of real images through the adversarial loss. Follow-up studies carry out various attempts while maintaining encoder-decoder structures with discriminators, presented in the work of context encoders. Among them, the attention module is exploited to generate realistic textures based on non-local search [15,16,17,18]. This module allows searching non-local regions to utilize information among similar structures in the image and updates pixel values of the previously estimated missing areas. Some studies perform structural restoration before the inpainting to clarify the boundary of the missing hole [16,2,19].
Although, recent neural approaches trained in a supervised manner with a large external training dataset for the inpainting task have shown promising and satisfactory results, they fail to fully utilize the information within given test images at the test phase. We propose a new learning approach to utilize internal statistics, available within the given test input image, such as patch recurrence, to overcome this limitation. Patch recurrence is a property that many similar patches are existing within a single natural image and numerous image restoration techniques utilize recurring patches to improve the restoration quality [20,21,22]. In particular, there are several attempts to train the restoration networks without using the ground truth clean images but train the network using the internal statistics only [23,24,25], and these self-supervised approaches have shown promising results. However, these self-supervised approaches fail to exert the power of deep learning through the large external datasets.
We demonstrate that inpainting networks can be trained in a self-supervised manner for the first time and develop a new learning algorithm that combines supervised and self-supervised approaches to benefit from both the external dataset and the given input image in this work. This approach improves the performance of existing inpainting networks by simply updating network parameters using the internal information (i.e., repeating structure/texture and color distribution) available from the given input image at test time. Specifically, we use parameters of fully pretrained inpainting networks with the large external dataset as initial values. We then fine-tune the network parameters in a self-supervised manner at the test stage. In particular, similar patches in a given input image are implicitly collected and can be utilized during the fine-tuning phase because of their repeated exposure to the network to improve the quality of output images, as shown in Figure 1. Note that broken lines and crushed edges of the initial-result images are clearly restored and the original shapes are properly predicted with the proposed fine-tuning algorithm. Our learning method is not restricted to specific network architecture and can be applied to various conventional networks. Moreover, our approach can easily upgrade the parameters of the conventional inpainting networks without changing their original architectures.
A simple and effective test-time parameter update method that upgrades inpainting networks by combining selfsupervision and supervision is proposed in this study. The main contributions of this study are summarized as follows: • A novel self-supervised fine-tuning algorithm that automatically takes advantage of patch recurrence within the test image is proposed.
• Superior inpainting results on benchmark datasets are achieved by utilizing both internal and external data.
• Our approach can be applied to several inpainting models without modifying their original network architectures or loss functions.

Related Work
We review numerous traditional approaches and recent learning-based methods for image inpainting as well as selfsupervised learning approaches for image restoration relevant to the proposed approach in this section.

Image inpainting
Traditional methods for image inpainting can be divided into the two categories of diffusion-and patch-based methods. Diffusion-based methods progressively propagate information into missing areas from the surrounding area [8,9,10,26], but only use local information for restoration. These approaches are effective when the missing hole is small, but creating meaningful content is difficult if the hole is large. Patch-based methods select highly similar patches within the given input image to fill the missing areas [27,12,28]. These methods can take advantage of not only local information but also non-local information and can fill the relatively large hole with meaningful content. PatchMatch [4] utilizes the fast nearest-neighbor field algorithm to reduce high computational cost and find patches similar to targets, and uses natural coherence in imagery to propagate acceptable patch match results to surrounding areas. Patch-based methods struggle to create unique content, such as human faces or complex scenes, because they need to meet the assumption that several patches similar to the missing hole in the image are available.
Deep learning methods have successfully been used in many computer vision tasks, and several learning-based approaches have been proposed for inpainting. Context encoder [13] introduces an encoder-decoder network based on convolutional neural network (CNN) to extract and restore semantics of the image and employs GAN [14] architecture to generate details. However, the reduction of resolution during encoding results in the blurry output image. Hence, Iizuka et al. [29] removed the pooling layer, employed dilated convolutions [30] in the bottleneck layer, and added a local discriminator that focuses on the missing area into a global discriminator, which assesses the entire image to maintain continuity in and out of the hole. Yu et al. [15] analyzed that CNN creates visual artifacts because of the inability to exploit image contents far from the hole. Thus, a contextual attention module is proposed to overcome this limitation. The contextual attention module scores the similarity of background patches with pixels of the missing area which are estimated in advance by a coarse network trained with L 1 loss. Then, patches with high similarity scores are used to reconstruct the missing area. Generative multi-column CNN [3] uses parallel networks with different receptive field sizes to prevent the propagation of errors from the coarse network to the fine network because networks are connected in series and utilizes spatial-variant loss to handle boundary consistency by giving different weights depending on the distance from the missing area. Liu et al. [31] proposed the use of partial convolution to learn various forms of masks. Previous methods were unutilized in practical cases where images were damaged in various forms because they were typically learned using a mask with a square hole. Partial convolution updates the mask during inference and renormalizes weights of the masked area every layer to propagate only valid information of the input. In a similar approach by Yu et al. [1], gated convolution automatically finds a proper mask for the input and considers the gating value calculated across a spatial location of every layer and channel. EdgeConnect [2] introduces a two-stage inpainting network model that first generates edge maps with a canny edge detector [32] and then recovers the missing region. However, this model is limited by its inability to utilize additional useful information, such as image color. Instead of the edge map, StructureFlow [16] uses a smooth image with preserved strong edges obtained via RTV [33] instead of the canny edge map and employs appearance flow [34] to achieve attention effect for realistic texture.

Self-supervised learning for restoration
The self-supervised learning-based approach for image restoration primarily refers to the training of the neural networks for the restoration of corrupted images using only information from the images themselves without a corresponding clean image. Lehtinen et al. [35] proposed the noise-to-noise approach for image denoising and demonstrated that the denoising network can be trained without a clean image using a pair of two noisy images. On this basis, Krull et al. [24] and Baston and Royer [25] introduced the self-supervised denoising approach which trains the network parameters at test time without using a ground-truth clean image. For the super-resolution task, zero-shot SR (ZSSR) [23] ] trains a small size of image-specific CNN during test time and utilizes internal statistics such as patch-recurrence within a given input image for SR. However, these methods can only utilize internal data because they train the network from random scratch at the test time.
The self-supervised approach is applied to image inpainting in this study to train (fine-tune) parameters for networks during test time by exploiting internal statics of a given masked input image. Our method exploits the advantage of both external and internal data by fine-tuning the fully pretrained network.

Proposed Method
In this section, we introduce a simple and effective self-supervised learning approach to adapt the parameters of pretrained inpainting networks to the given specific input test image.

Internal statistics for inpainting
The internal information available within the input image is very important for the image inpainting task and must be considered to generate semantically consistent and realistic results when filling missing areas.
Images generally demonstrate the property of patch recurrence, which enables the repeated presentation of many identical or similar patches in an image. The information of similar patches inside the image is used in past patch-based methods [27,12,4,28] and current deep-learning methods with attention mechanism [15,16]. Patch-based approaches find similar patches within the target image to fill the hole and paste them. By comparison, contextual attention [15] and appearance flow [16] in deep-learning approaches find similar patches/features in the given input image corresponding to the target area and resulting attention maps are used to complete the texture within the missing area. These methods improve the inpainting results by exploiting the internal information during test time, but are inefficient because they require high computational costs to measure similarity among features at every pixel location. Specifically, M N comparisons are required when the number of pixel values within missing and non-missing areas is N and M , respectively, but only a few matches of them produce meaningful high-similarity scores.
Therefore, we introduce a new approach that can fully utilize recurring patches in the given input image during the test phase without explicit similarity comparisons, and also present a new fine-tuning method that enhances the quality of inpainting results of existing models in a self-supervised manner.

Restore from Restored for Inpainting
Networks in existing learning-based image inpainting methods learn how to reconstruct the missing area using a large amount of training images during the training phase. However, the input image given at the test stage may include unique information, such as structures, textures, and colors, which remain ignored in the training dataset although a large external dataset is available for training. In this case, the pretrained network may generate an incompatible output with the specific pattern in the background. Therefore, we propose a new inpainting approach to exploit the specific information available within the input test image in this work by utilizing multiple similar patches in the input image and further improve the pretrained network by adapting network parameters to the specific input image. Figure 2 illustrates how self-similar patches can be used to fine-tune the network parameters and improve the performance of conventional inpainting networks. We first obtain an initial inpainting result I pred by applying the fully pretrained EdgeConnect [2] from an input imageĨ distorted with a squared mask (Figure 2(a)). We then generate a newly masked imageĨ pred to utilize repeating structures of the given input image for inpainting by removing a self-similar patch in I pred using the squared mask. By minimizing the MSE between I pred and the newly masked imagẽ I pred , we fine-tune network the parameters. Accordingly, we can update network parameters to the input image and can improve the pretrained inpainting network (Figure 2(b)). Note that we adapt network parameters without using the ground-truth image during fine-tuning. Moreover, we can also fine-tune the network by corrupting the initially restored image randomly (Figure 2(c)). Specifically, we generate multiple training images for fine-tuning by corrupting I pred using random masks, then fine-tune and upgrade the network by minimizing MSE between these newly corrupted images and the initial inpainting result I pred . By doing so, we demonstrate that we can adapt network parameters without explicitly searching self-similar patches within the input. Notably, traditional methods require the use of dedicated algorithms and specialized network modules for explicit searching in the process of finding similar patches [4,27,12,28]. By comparison, we utilize self-similar patches using the randomized masking scheme during the test phase without changing the original network architecture and requiring any additional algorithm and/or network modules for explicit searching.

Overall Flow
First, we start the learning process with the initially restored image I pred similar to [36] by using the fully pretrained network as follows: where θ 0 denotes the fully pretrained parameters of the baseline inpainting network f andĨ denotes the masked input image. A binary map M denotes a given input mask that corresponds to the input image, where missing and other areas are represented by 1 and 0, respectively.
Second, we acquire a training dataset using the initially restored image I pred . We generate a new and randomly corrupted masked image at the i th fine-tuning iteration as follows: where M i denotes the randomly generated binary mask and represents the element-wise multiplication. The newly synthesized masked imageĨ pred and I pred become the input and target of our training dataset, respectively. We generate the restored image I pred(θ) from the new masked imageĨ pred using the pretrained network.
Third, we compute gradient values with respect to network parameters using the predefined loss function for pretraining the baseline inpainting network and then update network parameters. The loss is calculated using the difference between I pred and I pred(θ) . Notably, if the baseline inpainting network includes GAN architecture, the discriminator of GAN architecture computes the adversarial loss by using the initially restored image I pred as the real sample rather than the ground-truth clean image.
Specifically, for the pixel-wise loss (i.e., reconstruction loss), we can use the L2 loss function as follows: In the proposed reconstruction loss, we exclude the part of the original mask M since we can improve the performance by making this slight modification (see Sec. 4.2). Note that, we can use any conventional reconstruction losses beyond the L2 loss (e.g., L1).
Moreover, when considering additional (perceptual) losses, such as VGG and and adversarial losses, we do not make any modification. We employ the original function used to train the baseline inpainting networks, and the expression is loss 2 (θ). Thus, our overall loss function that updates parameters of the pretrained network is expressed as follows: We repeat these steps T times, which are the minimum number of iterations determined experimentally to achieve better results than I pred . Finally, the network parameter θ is updated to θ * and we obtain the fine-tuned image f θ * (Ĩ, M ) with the input imageĨ and original mask M using the fine-tuned network f θ * .
First, we use officially available and fully pretrained network parameters on the Places2 dataset [37] for each network and fine-tune the pretrained parameters via the proposed approach in Algorithm. 1. We evaluate the performance of the proposed algorithm on test sets of conventional benchmark datasets, such as Places2 [37] and Urban100 [22].
We generate training inputs to fine-tune models at the test time by adding random free-form masks [1] to the initially restored image I pred . To be specific, each train input includes holes that cover around 20% to 40% of the entire image. We utilize the same optimizer and loss function used in each pretrained models.

Effect of the number of iterations on image quality of fine-tuned results
We set up the fine-tuning iteration that maximizes the quantitative and qualitative results of images for each model and dataset. We will show changes in the image quality depending on the number of fine-tuning iterations through the experiment conducted with the Urban100 dataset. Figure 4 illustrates the results of the fine-tuned network with different numbers of iterations for each pretrained model. The structure of the results is gradually recovered as the fine-tuning progresses, and the results are close to the ground truth when it comes to the optimal iteration. Moreover, Figure 5 shows the trend of Fréchet inception distance (FID) scores and peak SNR (PSNR) values with a fine-tuning process for each model. We observe that training longer than the optimal number of iterations can lead to overfitting depending on the model. The FID score can be an important criterion for determining the optimal number of iterations because of its higher sensitivity to overfitting than the PSNR value.
For the remaining experiments, we use the fixed number of iterations chosen in this manner for fine-tuning. Specifically, the number of fine-tuning iterations is determined according to inpainting models and datasets. At the test phase, we fine-tune 500, 1000, and 200 iterations on the Places2 test dataset and 700, 1000, and 250 iterations on the Urban100 dataset for each pretrained model (i.e., GatedConv, EdgeConnect, and GMCNN).

Exclude or include the original mask in the loss calculation
As mentioned in Section 3.3, we excluded the original mask part when calculating the loss during the finetuning step. We perform ablation experiments exploiting the entire baseline image which includes the generated part for the fine-tuning and compare the results with the findings of our method. Figure 7 shows the PSNR and SSIM values according to the fine-tuning iteration when parts of the original mask are included/excluded in the loss calculation. The results without the original mask demonstrate a higher score in all steps than the findings with the mask. Figure 6 presents the comparison of experiments. Parts of the baseline image generated from the pretrained network sometimes demonstrate mismatched colors with its surroundings. As a result, colors of generated and surrounding parts are slightly mismatched when these parts are included in the loss calculation at the fine-tuning step, whereas the color of both parts demonstrates satisfactory matching when the mask is excluded.

Quantitative results
We quantitatively evaluate the performance of fine-tuned networks in terms of PSNR and structural similarity index (SSIM) to compare the inpainting results objectively. Moreover, we provide the quantitative comparison results to measure perceptual quality, which calculates how plausible the produced image looks from the human perspective. These comparisons are necessary because the inpainting task guesses not only the image before it was damaged but also ultimately generates a visually appealing image for humans.   For comparisons, we use learned perceptual image patch similarity (LPIPS) [38] and FID [39]. LPIPS compares deep features of images extracted from the image classifier based on a CNN architecture, such as VGG [40] and AlexNet [41]. We use AlexNet in this study. FID measures Wasserstein distance (Fréchet distance) between distributions of real and generated data. The mean and covariance of each distribution are calculated through the pretrained Inception-V3 [42] model. Tables 2 and 1 list the quantitative results. Table 2 shows the comparison of the quantitative results from each fine-tuning iteration on the Places2 dataset. One thousand images from the Places2 test dataset are used for the evaluation. Overall metric values increase compared with the results of the baseline models (equivalent to fine-tuning 0 iteration, T = 0) due to the fine-tuning. Table 1 presents the improvements between the results of pretrained models and the findings after fine-tuning on the Urban100 dataset. Although the Urban100 dataset is not used to train pretrained networks, it shows considerable improvement due to fine-tuning. This finding proves our fine-tuning methods improve the results although the test input image has a different distribution from the dataset used in pretraining.

Qualitative results
We compare the qualitative results between the initially restored results by the pretrained models and our results by fine-tuning the pretrained models. Figure 8 shows visual results on the Places2 dataset. Note that, the results of pretrained models show the generated part mismatch the other parts and are crushed. By comparison, the results of our fine-tuned models restore the input image more naturally. Furthermore, the results are significantly improved if  recurring patches, such as windows or stairs, exist in the image, because the network will likely learn the correct answers using many similar patches. Figure 9 shows the results of the Urban100 dataset. The Urban100 dataset demonstrates improved results for structural restoration because its images consist of many repetitive structures, such as buildings and windows.

Run-time
The experiment with RTX 2080Ti lasted 0.15, 1, 0.4 seconds for 1 iteration per 256 x 256 images with the GatedConv, EdgeConnect, and GMCNN network models.

Conclusion
A new self-supervision-based inpainting algorithm that allows the adaptation of fully pretrained network parameters during the test stage is proposed in this work. We utilize self-similar patches within the given input test image to fine-tune the network without using the ground-truth clean image and elevate the performance of networks by combining internal and large external information. We can easily fine-tune baseline networks and significantly improve the inpainting performance over baselines by optimizing slightly modified loss functions, which are used to train baseline networks with conventional optimizers. The proposed method achieves state-of-the-art inpainting results on conventional inpainting datasets, and extensive experimental results demonstrate the performance of the proposed algorithm.  : Qualitative comparison of results on the Urban100 dataset before and after fine-tuning pretrained models. The first two columns show the ground-truth and masked input image. The next two columns represent image pairs, where the left is the baseline image obtained via the pretrained inpainting models and the right is the result images by our fine-tuning algorithm. The models are listed in the order of GatedConv, EdgeConnect, and GMCNN.