{'fraction_non_alphanumeric': 0.03455365811868182, 'fraction_numerical': 0.011708264321206236, 'mean_word_length': 4.840560373582388, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Image Inpainting via Generative Multi-column Convolutional Neural Networks

In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.


Spatial-variant Constraints
Another important issue is that inpainting can take multiple candidates to fill holes. Thus, optimal results should be constrained in a spatially variant way -pixels close to area boundary are with few choices, while the central part can be less constrained. In fact, adversarial loss has already been used in recent methods [18,25,24,9,26] to learn multi-modality. Various weights are applied to loss [18,25,26] for boundary consistency. In this paper, we design a new spatial-variant weight to better handle this issue.
The overall framework is a Generative Multi-column Convolutional Neural Network (GMCNN) for image inpainting. The multi-column structure [3,27,1] is used since it can decompose images into components with different receptive fields and feature resolutions. Unlike multi-scale or coarse-tofine strategies [24,12] that use resized images, branches in our multi-column network directly use full-resolution input to characterize multi-scale feature representations regarding global and local information. A new implicit diversified Markov random field (ID-MRF) term is proposed and used in the training phase only. Rather than directly using the matched feature, which may lead to visual artifacts, we incorporate this term as regularization.
Additionally, we design a new confidence-driven reconstruction loss that constrains the generated content according to the spatial location. With all these improvements, the proposed method can produce high quality results considering boundary consistency, structure suitability and texture similarity, without any post-processing operations. Exemplar inpainting results are given in Figure 1.

Related Work
Exemplar-based Inpainting Among traditional methods, exemplar-based inpainting [7,8,4,21,10,11,2] copies and pastes matching patches in a pre-defined order. To preserve structure, patch priority computation specifies the patch filling order [4,7,8,21]. With only low-level information, these methods cannot produce high-quality semantic structures that do not exist in examples, e.g., faces and facades.
CNN Inpainting Since the seminal context-encoder work [18], deep CNNs have achieved significant progress. Pathak et al. proposed training an encoder-decoder CNN and minimizing pixel-wise reconstruction loss and adversarial loss. Built upon context-encoder, in [9], global and local discriminators helped improve the adversarial loss where a fully convolutional encoder-decoder structure was adopted. Besides encoder-decoder, U-net-like structure was also used [23].
Yang et al. [24] and Yu et al. [26] introduced coarse-to-fine CNNs for image inpainting. To generate more plausible and detailed texture, combination of CNN and Markov Random Field [24] was taken as the post-process to improve inpainting results from the coarse CNN. It is inevitably slow due to iterative MRF inference. Lately, Yu et al. conducted nearest neighbor search in deep feature space [26], which brings clearer texture to the filling regions compared with previous strategies of a single forward pass.

Our Method
Our inpainting system is trainable in an end-to-end fashion, which takes an image X and a binary region mask M (with value 0 for known pixels and 1 otherwise) as input. Unknown regions in image X are filled with zeros. It outputs a complete imageŶ. We detail our network design below.   Figure 2: Our framework.

Network Structure
Our proposed Generative Multi-column Convolutional Neural Network (GMCNN) shown in Figure 2 consists of three sub-networks: a generator to produce results, global&local discriminators for adversarial training, and a pretrained VGG network [20] to calculate ID-MRF loss. In the testing phase, only the generator network is used.
The generator network consists of n (n = 3) parallel encoder-decoder branches to extract different levels of features from input X with mask M, and a shared decoder module to transform deep features into natural image spaceŶ. We choose various receptive fields and spatial resolutions for these branches as shown in Figure 2, which capture different levels of information. Branches are denoted as {f i (·)} (i ∈ {1, 2, ..., n}), trained in a data driven manner to generate better feature components than handcrafted decomposition.
Then these components are up-sampled (bilinearly) to the original resolution and are concatenated into feature map F . We further transform features F into image space via shared decoding module with 2 convolutional layers, denoted as d(·). The output isŶ = d(F ). Minimizing the difference betweenŶ and Y makes {f i (·)} i=1,...,n capture appropriate components in X for inpainting. d(·) further transforms such deep features to our desired result. Note that although f i (·) seems independent of each other, they are mutually influenced during training due to d(·).
Analysis Our framework is by nature different from commonly used one-stream encoder-decoder structure and the coarse-to-fine architecture [24,26,12]. The encoder-decoder transforms the image into a common feature space with the same-size receptive field, ignoring the fact that inpainting involves different levels of representations. The multi-branch encoders in our GMCNN contrarily do not have this problem. Our method also overcomes the limitation of the coarse-to-fine architecture, which paints the missing pixels from small to larger scales where errors in the coarse-level already influence refinement. Our GMCNN incorporates different structures in parallel. They complement each other instead of simply inheriting information.

ID-MRF Regularization
Here, we address aforementioned semantic structure matching and computational-heavy iterative MRF optimization issues. Our scheme is to take MRF-like regularization only in the training phase, named implicit diversified Markov random fields (ID-MRF). The proposed network is optimized to minimize the difference between generated content and corresponding nearest-neighbors from ground truth in the feature space. Since we only use it in training, complete ground truth images make it possible to know high-quality nearest neighbors and give appropriate constraints for the network.
To calculate ID-MRF loss, it is possible to simply use direct similarity measure (e.g. cosine similarity) to find the nearest neighbors for patches in generated content. But this procedure tends to yield smooth structure, as a flat region easily connects to similar patterns and quickly reduces structure variety, as shown in Figure 3(a). We instead adopt a relative distance measure [17,16,22] to model the relation between local features and target feature set. It can restore subtle details as illustrated in Figure 3(b).
Specifically, letŶ g be the generated content for the missing regions,Ŷ L g and Y L are the features generated by the L th feature layer of a pretrained deep model. For neural patches v and s extracted fromŶ L g and Y L respectively, the relative similarity from v to s is defined as where µ(·, ·) is the cosine similarity. r ∈ ρ v (Y L ) means r belongs to Y L excluding v. h and are two positive constants. If v is like s more than other neural patches in Φ L (Y), RS(v, s) turns large.
Next, RS(v, s) is normalized as Finally, with Eq. (2), the ID-MRF loss betweenŶ L g and Y L is defined as where Z is a normalization factor.
closer to s compared with other neural patches inŶ L g . In the extreme case that all neural patches in Y L g are close to one patch s, other patches r have their On the other hand, when the patches inŶ L g are close to different candidates in Y L , each r in Y L has its unique nearest neighbor inŶ L g . The resulting max v∈Ŷ L g RS(v, r) is thus big and L M (L) becomes small. We show one example in the supplementary file. From this perspective, minimizing L M (L) encourages each v inŶ L g to approach different neural patches in Y L , diversifying neighbors, as shown in Figure 3(b).
An obvious benefit for this measure is to improve the similarity between feature distributions inŶ L g and Y L . By minimizing the ID-MRF loss, not only local neural patches inŶ L g find corresponding candidates from Y L , but also the feature distributions come near, helping capture variation in complicated texture.
Our final ID-MRF loss is computed on several feature layers from VGG19. Following common practice [5,14], we use conv4_2 to describe image semantic structures. Then conv3_2 and conv4_2 are utilized to describe image texture as More Analysis During training, ID-MRF regularizes the generated content based on the reference. It has the strong ability to create realistic texture locally and globally. We note the fundamental difference from the methods of [24,26], where nearest-neighbor search via networks is employed in the testing phase. Our ID-MRF regularization exploits both reference and contextual information inside and out of the filling regions, and thus causes high diversity in inpainting structure generation.

Information Fusion
Spatial Variant Reconstruction Loss Pixel-wise reconstruction loss is important for inpainting [18,25,26]. To exert constraints based on spatial location, we design the confidence-driven reconstruction loss where unknown pixels close to the filling boundary are more strongly constrained than those away from it. We set the confidence of known pixels as 1 and unknown ones related to the distance to the boundary. To propagate the confidence of known pixels to unknown ones, we use a Gaussian filter g to convolve M to create a loss weight mask M w as where g is with size 64 × 64 and its standard deviation is 40.
where G([X, M]; θ) is the output of our generative model G, and θ denotes learn-able parameters.
Compared with the reconstruction loss used in [18,25,26], ours exploits spatial locations and their relative order by considering confidence on both known and unknown pixels. It results in the effect of gradually shifting learning focus from filling border to the center and smoothing the learning curve.
Adversarial Loss Adversarial loss is a catalyst in filling missing regions and becomes common in many creation tasks. Similar to those of [9,26], we apply the improved Wasserstein GAN [6] and use local and global discriminators. For the generator, the adversarial loss is defined as whereX = tG([X, M]; θ) + (1 − t)Y and t ∈ [0, 1].

Final Objective
With confidence-driven reconstruction loss, ID-MRF loss, and adversarial loss, the model objective of our net is defined as where λ adv and λ mrf are used to balance the effects between local structure regularization and adversarial training.

Training
We train our model first with only confidence-driven reconstruction loss and set λ mrf and λ adv to 0s to stabilize the later adversarial training. After our model G converges, we set λ mrf = 0.05 and λ adv = 0.001 for fine tuning until converge. The training procedure is optimized using Adam solver [13] with learning rate 1e − 4. We set β 1 = 0.5 and β 2 = 0.9. The batch size is 16.

Experimental Settings
We train our models on the training set and evaluate our model on the testing set (for Paris street view) or validation set (for Places2, ImageNet, CelebA, and CelebA-HQ

Qualitative Evaluation
As shown in Figures 8 and 10, compared with other methods, ours gives obvious visual improvement on plausible image structures and crisp textures. The more reasonably generated structures mainly stem from the multi-column architecture and confidence-driven reconstruction loss. The realistic textures are created via ID-MRF regularization and adversarial training by leveraging the contextual and corresponding textures.
In Figures 9, we show partial results of our method and CA [26] on CelebA and CelebA-HQ face datasets. Since we do not apply MRF in a non-parametric manner, visual artifacts are much reduced. It is notable that finding suitable patches for these faces is challenging. Our ID-MRF regularization remedies the problem. Even the face shadow and reflectance can be generated as shown in Figure 9. Also, our model is trained with arbitrary-location and -size square masks. It is thus general to be applied to different-shape completion as shown in Figures 10 and 1. More inpainting results are in our project website.

Quantitative Evaluation
Although the generation task is not suitable to be evaluated by peak signal-to-noise ratio (PSNR) or structural similarity (SSIM), for completeness, we still give them on the testing or validation sets of four used datasets for reference. In ImageNet, only 200 images are randomly chosen for evaluation since MSNPS [24] takes minutes to complete a 256 × 256 size image. As shown in Table 1, our method produces decent results with comparable or better PSNR and SSIM.
We also conduct user studies as shown in Table 2. The protocol is based on large batches of blind randomized A/B tests deployed on the Google Forms platform. Each survey involves a batch of 40 pairwise comparisons. Each pair contains two images completed from the same corrupted input by two different methods. There are 40 participants invited for user study. The participants are asked to select the more realistic image in each pair. The images are all shown at the same resolution (256×256). The comparisons are randomized across conditions and the left-right order is randomized. All images are shown for unlimited time and the participant is free to spend as much time as desired on each pair. In all conditions, our method outperforms the baselines.

Ablation Study
Single Encoder-Decoder vs. Coarse-to-Fine vs. GMCNN We evaluate our multi-column architecture by comparing with single encode-decoder and coarse-to-fine networks with two sequential encoder-decoder (same as that in [26] except no contextual layer). The single encoder-decoder is just the same as our branch three (B3). To minimize the influence of model capacity, we triple the filter sizes in the single encoder-decoder architecture to make its parameter size as close to ours as possible. The loss for these three structures is the same, including confidence-driven reconstruction loss, ID-MRF loss, and WGAN-GP adversarial loss. The corresponding hyper-parameters are the same. The testing results are shown in Figure 4. Our GMCNN structure with varied receptive fields in each branch predicts reasonable image structure and texture compared with single encoder-decoder and coarse-to-fine structure. Additional quantitative experiment is given in Table 3, showing the proposed structure is beneficial to restore image fidelity.
Varied Receptive Fields vs. Fixed Receptive Field We then validate the necessity of using varied receptive fields in branches. The GMCNN with the same receptive field in each branch turns to using 3 identical third Branches in Figure 2 with filter size 5 × 5. Figure 4 shows within the GMCNN structure, branches with varied receptive fields give visual more appealing results. Spatial Discounted Reconstruction Loss vs. Confidence-Driven Reconstruction Loss We compare our confidence-driven reconstruction loss with alternative spatial discounted reconstruction loss [26]. We use a single-column CNN trained only with the losses on the Paris street view dataset. The testing results are given in Figure 5. Our confidence-driven reconstruction loss works better.
With and without ID-MRF Regularization We train a complete GMCNN on the Paris street view dataset with all losses and one model that does not involve ID-MRF. As shown in Figure  6, ID-MRF can significantly enhance local details. Also, the qualitative and quantitative changes are given in Table 4 and Figure 7 about how λ mrf affects inpainting performance. Empirically, λ mrf = 0.02 ∼ 0.05 strikes a good balance.

Conclusion
We have primarily addressed the important problems of representing visual context and using it to generate and constrain unknown regions in inpainting. We have proposed a generative multi-column neural network for this task and showed its ability to model different image components and extract multi-level features. Additionally, the ID-MRF regularization is very helpful to model realistic texture with a new similarity measure. Our confidence-driven reconstruction loss also considers spatially variant constraints. Our future work will be to explore other constraints with location and content.
Limitations Similar to other generative neural networks [18,24,26,25] for inpainting, our method still has difficulties dealing with large-scale datasets with thousands of diverse object and scene categories, such as ImageNet. When data falls into a few categories, our method works best, since the ambiguity removal in terms of structure and texture can be achieved in these cases.