{'fraction_non_alphanumeric': 0.028367371504233036, 'fraction_numerical': 0.009388917867030454, 'mean_word_length': 4.864470588235294, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GLocal: Global Graph Reasoning and Local Structure Transfer for Person Image Generation

In this paper, we focus on person image generation, namely, generating person image under various conditions, e.g., corrupted texture or different pose. To address texture occlusion and large pose misalignment in this task, previous works just use the corresponding region's style to infer the occluded area and rely on point-wise alignment to reorganize the context texture information, lacking the ability to globally correlate the region-wise style codes and preserve the local structure of the source. To tackle these problems, we present a GLocal framework to improve the occlusion-aware texture estimation by globally reasoning the style inter-correlations among different semantic regions, which can also be employed to recover the corrupted images in texture inpainting. For local structural information preservation, we further extract the local structure of the source image and regain it in the generated image via local structure transfer. We benchmark our method to fully characterize its performance on DeepFashion dataset and present extensive ablation studies that highlight the novelty of our method.


Introduction
Person image generation aims to generate realistic person image with target pose while keeping the source appearance unchanged, which is a popular task in the community of computer vision with applications like image editing and image animation. Perennial research efforts have contributed to impressive performance gain on challenging benchmark.
Essentially, the person image generation involves the deformation of human body in 3D space, which poses an illposed problem given just the 2D image and pose inputs. The limitation of 2D modeling nature leads to tough challenges due to the following difficulties: (i) the prediction of invisible parts in the source caused by occlusion or corruption as shown in Figure 1, (ii) retaining the spatial structure of visible source texture. The exploration of these issues has contributed to the development and progress of this field. inputs. Thus it is unable to utilize the appearance correspondence between the input and target image. To achieve better rearrangement of source appearance, warping-based (Dong et al. 2018;Siarohin et al. 2018;Tang et al. 2021a;Li, Huang, and Loy 2019;Liu et al. 2019;Ren et al. 2020) methods have been proposed. However, these operations struggle in recovering structural details by solely relying on the point-wise mapping and incur a defect when dealing with unmatched occlusion regions. Recent works (Men et al. 2020;Zhang et al. 2021) apply semantic normalization techniques such as AdaIN (Men et al. 2020), SEAN (Zhu et al. 2020) to the semantic details restoration in the target. However, the non-discriminatory treatment of the occluded and non-occluded regions introduces misleading information and its capability to preserve structural details is not fully exploited. This paper presents GLocal, a scheme to preserve the semantic and structural information with Global Graph Rea-soning (GGR) and Local Structure Transfer (LST), respectively. Inspired by the fact that the invisible area estimation can be facilitated by visible area (e.g. if the hand is unseen in the source image, its color can still be estimated by the neck skin), we manage to estimate the occluded region style by modeling the internal relationships among different semantic regions. The relationships are built with a graph architecture based on the linkage of each part, whose graph nodes are filled with per-region styles. Thereby the invisible area's style representation can be inferred by our GGR. After acquiring the global statistics of the source, we seek to reproduce source's local structure with LST since local structure can indicate how the image details are organized and is better than isolated points. We achieve this by predicting style parameters of the source from local correlation map and transferring the structural information of the source style to the generation features with LocConv. Our contributions can be summarized as follows: • We propose GGR module to estimate the occluded region style with global reasoning in person image generation, which conveys visible style information to the invisible area along with the graph structure according to the connectivity of human body. • We design LST module to extract local structure and transfer it into the generated result, which sharpens the generation result and leads to better detail conservation. • Extensive experiments show that our method achieves superior performance on the challenging DeepFashion dataset both qualitatively and quantitatively.

Related Work Person Image Generation
Person image generation is a valuable branch of the mainstream image generation field, which focuses on the humanspecific generation task. As a milestone, (Ma et al. 2017) first shows that conditional GAN (cGAN) can generate desired person images. Extending this idea, several following works seek to improve cGAN's performance with techniques such as unsupervised training (Pumarola et al. 2018), cycle training (Tang et al. 2020a), coarse-to-fine strategy (Wei et al. 2020), and progressive training , etc. However, these methods ignore the vanilla conditional GAN's incapability to model the relationship on the large pose misalignment. To better achieve spatial rearrangement, (Siarohin et al. 2018) decomposes the translation between different poses into local affine transformations. Approaches such as (Li, Huang, and Loy 2019;Wei et al. 2021;Tang et al. 2021b;Liu et al. 2019;Ren et al. 2020) turn to estimate the appearance flow to align the source appearance with the target. Although warping the source can alleviate the aforementioned problem, they cannot deal with the occluded-region estimation for their false assumption that pixel-to-pixel warping transformation exists between source and target. Recently, emerging modulationbased methods Lv et al. 2021) adopt the two-stage pipeline, which first predict the segmentation map to extract region-wise image styles, and then use them to guide region synthesis with corresponding style injection. Unfortunately, this strategy cannot extract appropriate styles for occluded regions, and thus still fails to synthesize them. Different from previous methods, our global graph reasoning module can mitigate the occluded texture generation issues and achieve better structural preservation of the source in the target.

Semantic Image Generation
Conditioned on a semantic segmentation map, semantic image generation is a special form of general image generation, whose solution is commonly based on style control. Style is mainly defined as the statistical property of the image feature which can be spatial varying (Park et al. 2019), region adaptive (Zhu et al. 2020), or class specific (Tan et al. 2021b,a).
To transfer the style from one to another, they model the style as the modulation scale/shift parameters. Specifically, SPADE (Park et al. 2019) predicts spatial-varying affine parameters which modulates reference image features in different locations and obtains more dedicated style injection and high fidelity generation results. Later, SEAN (Zhu et al. 2020) enforces the per-region style extraction and modulation. Our method stems from this scheme and injects the desired style based on whether it is occluded or not, which achieves more precise semantic manipulation.

Method Overview
We propose a novel framework GLocal for person image generation which is divided into three stages: semantic prediction, global graph reasoning, and local structure transfer. As shown in Figure 2, the semantic prediction network can supply semantic layout for per-region style extraction and injection in the global graph reasoning stage. The local structure transfer module further controls the generation process with local structure modeling and transferring.
Preliminary To align the source feature F s with the target F t and supply visibility guidance, we adopt the Intr-Flow (Li, Huang, and Loy 2019) to learn the appearance warping flow W and visibility map m vis by matching the 3D human body model. The visibility map can indicate whether it is visible or not in the target for each source position.

Semantic Prediction
Directly generating a person image solely from target pose map involves the estimation of human semantic and body details, which is troublesome since they are highly related to each other. To simplify the overall generation process, we turn to generate the target semantic map first, which is easier to estimate and can provide better semantic assistance for further texture detail complement. Our semantic prediction network adopts the vanilla pix2pix (Isola et al. 2017) architecture, which takes the source image I s , source pose map P s , and target pose map P t as inputs. The semantic map consists of 8 semantic classes, among them some classes occupying a large area comprise the majority of the loss, which leads to inefficient : An overview of our GLocal framework. We first predict the target segmentation map S g from Semantic Prediction. Then we get the warping flow W and visibility map m vis from a pretrained 3D flow estimation network (Li, Huang, and Loy 2019). We mark the visible and invisible positions with green and red in m vis respectively. After fusing the encoded target and warped source features, we can inject the source style into relevant regions within GGR module precisely. Finally, in the LST module, we transfer the local structure from source to generated result and send it into the decode to reconstruct the final image.
learning for small parts (e.g., shoes). Considering this, we propose to utilize the focal loss (Lin et al. 2017) to alleviate the class imbalance problem. Mathematically, the focal loss in class-balanced semantic prediction is formulated as: (1) where S t ∈ {1, 0} specifies the ground-truth target segmentation map and S g ∈ [0, 1] denotes the predicted probability for the class with label S t = 1. η acts as the tunable focusing parameter and thus the scaling factor (1 − p t ) η can effectively focus on the hard classes, e.g. shoes or hands.

Global Graph Reasoning
As exemplified in Figure 3, to extract the style codes of the source for each region, we perform per-region average pooling on the encoded source features according to the source segmentation map S s and obtain the source's style code ST ∈ R N ×512 , where N denotes the number of semantic regions. To distinguish the occluded and non-occluded area for each region, we calculate the occlusion map m oc with predicted S g and visibility map m vis by element-wise product. For non-occluded area, we inject each style into corresponding region. Then we perform per-style convolution and broadcast the convolved style codes to target regions. However, for occluded area, directly generating with corresponding source region's style information may introduces irrationality. Thus we need to supply more accurate style representation for occluded area estimation with global style reasoning.
Graph Modeling for Source Styles Since some body parts generally share similar appearance characteristics, e.g., the neck and the hand should be highly analogous in color. so it is natural to aid the occlusion estimation with global graph style propagation along with the human body struc-ture. Given the encoded source feature style codes, we construct a relation graph with region-wise style code as graph nodes and natural connectivities in human body structure. Then the style feature nodes can be recurrently updated via graph propagation. Specifically, we construct a spatial graph G = (V, E) on the N regions to feature the inter-region connection, where the node set V = {v i | i = 1, . . . , N } represents the all per-region style vectors. These nodes are connected with edges E = {v i v j } according to the connectivity of human body structure. Then we perform spatial graph convolution to propagate the style information. The graph convolved node value at x can be written as: (2) where the sampling function p : and D is set to 1. The weighting function w (v i , v j ) in graph convolution allocates a specific weighting value to each sampled node according to the subset it belongs, unlike the 2D convolution convolves its pixels according to the fixed square order. As illustrated in Figure 4, we divide the neighbor set B (v i ) of node v i into three subsets with centrifugal distance comparison. The mapping from nodes to its subset label is defined as: where e i denotes the average distance from gravity center to node i. The convolution product of each subset is normalized by the balance term After reasoning the occlusion area with style code graph, we can advance to the generation of conditioning feature map, whose occluded regions are now filled with globally reasoned source style features. By performing convolution  and broadcasting on the conditional feature map, we obtain two sets of spatial-varying modulation parameters γ and β. They act as the scale and bias to modulate the normalized feature map

Local Structure Transfer
The GGR module can only capture the global statistics of the source and thus localized structure cannot be effectively preserved from source to target. As shown in Figure 5, to transfer the local structural context relationship, we first get local correlation map F c , which models the local structure with self-correlation layer. Then we predict the full convolution kernels f and bias b, since these kernels better distill local spatial structure. After aligning these parameters with the target by optimal transport, we perform LocConv to transfer the local structure of the source to the generation activations via convolution on the normalized target features.
Modeling Local Structure Intuitively, the local structure of the feature map can be represented by the adjoining patch correlation patterns (i.e. the relationship of one patch with its neighbors). In light of this, we extract the local structural representation of source features with a self-correlation layer that perform multiplicative patch comparisons around each source position. Formally, given the source feature map F s ∈ R c×H×W , our self-correlation layer calculates the correlation of two patches centered at i and its neighbors j ∈ N (i) via vector product, which is defined as where the Concat denotes the channel-wise concatenate. Note that for computational restriction, we just compute self-correlation c(i, j) with neighbors whose distance from i is less than d. After correlating the positions around i, we concatenate the correlations in channel to get the local structure representation F c ∈ R (2d+1) 2 ×H×W . To better distill the local structure statistics and assist the structural transferring, we predict the spatial-and channel-varying modulation values including the 3D filter f ∈ R H×W ×c×(k×k) and bias b ∈ R H×W ×c×1 from the local structure representation F c via point-wise convolutions.

Modulation Parameters Alignment
To further align the modulation parameters with the target, we introduce the Unbalanced Optimal Transport (UOT) (Zhan et al. 2021) mechanism to match the encoded source F s and generation features F o with calculated transport plan (T P ). To solve a optimal transport problem which aims to transform one collections of masses to another, we define the masses of source and generated features with dirac form: α = n i=1 α i δ si and β = n i=1 β i δ oi , where the s i and o i denote the positions of α i and β j in the F s and F o , respectively. Then the optimal transport problem can be formulated as: where the cost matrix C is formulated as C ij = 1 − s i ·ocj si ocj , giving the cost to move mass α i to β j . T P denotes the transport plan of which each element T P ij denotes the quantity of masses transported between α i to β j . With the transport plan, we can warp f and b to get the aligned modulation parametersf andb, which is formulated as: LocConv for Stucture Transfer Unlike 1 × 1 kernel size adopted in the conventional point-wise modulation (e.g., SPADE (Park et al. 2019)), the k×k square filter of LocConv allows for modulating the generation acitvations with the local structure in the neighborhood around point l : (i, j) of feature map F o . Then we can obtain generation feature map F g by modulating the aligned modulation filterf and biasb at F o in a spatial-and channel-varying way. The value of F g located at l : (i, j) is defined as: where µ Fo and σ Fo represent the channel-wise mean and standard deviation of the F o . Finally we can get the F g by enumerating positions in F o .

Learning Objectives
After the semantic prediction network is trained with our proposed L(S g ). The whole networks is then trained end-toend with source image I s , source pose P s , and target pose P t as inputs. We encourage the generated target-posed image I g to close in the groundtruth target image I t in image and perceptual level. Thus we introduce following learning objectives to guide the training process. Pixel-wise Loss. To generate more sharp images, we use L L1 to measure the pixel-wise fiderlity between the generated imageI g and the groundtruthI t in image pixel space. L L1 = ||I g − I t || 1 , (9) Perceptual Loss. Besides the pixel-wise loss, we also adopt the perceptual similarity measurement in VGG-19 (Simonyan and Zisserman 2014) feature space with perceptual loss.
where φ k represents the neuron response at k th layer extracted with a pretrained VGG-19 model. Adversarial Loss. Due to the great potential of GAN, we introduce the adversarial loss to encourage the generator to generate photo-realistic images in adversarial manner. The training objective for the discriminator D and generator G is caculated with: (11) Overall loss function. The final learning objective is incorporated with above-mentioned losses, which is defined as: where α Sg , α L1 , α perc , α adv are the trade-off weights.

Experiments
In this section, we first introduce implementation details including dataset and image quality evaluation metrics. Then we perform extensive experiments to illustrate the superiority over the prevalent methods and verify the effectiveness of our model.

Implementation Details
We select DeepFashion (Liu et al. 2016) to evaluate our method for its diversity among human identity and coverage of different clothes. The DeepFashion dataset contains 52712 high resolution images which show various clothing styles and poses. All images are resized to 256x176, which is adopted by many previous methods. Following the strategy of (Ren et al. 2020), we obtain image pairs and split them into 101966 for training and 8570 for testing without overlap. To measure generated results from different aspects, we choose Structural Index Similarity (SSIM) and Peak Signal to Noise Ratio (PSNR) for image-level similarity measurement which are sensitive to the image quality of imperceptibility. For perceptual evaluation, Inception Score (IS) and Fréchet Inception Distance (FID) are introduced to calculate the feature-level and distribution-based distance between the InceptionNet-encoded features. Figure 6 presents the qualitative comparisons between several state-of-the-art methods and our model, which demonstrates the superiority of our method in clothes structure recovery, photo-realistic texture rendering, and distinct person identities preservation. The LiquidGAN  will generate weird human images when accurate 3D human modeling is not available. Methods such as PoNA , XingGAN (Tang et al. 2020b), and GFLA (Ren et al. 2020) tend to generate unreasonable texture for lack of semantic guidance. Besides, the PISE  and SPGNet (Lv et al. 2021) overlook the inter-region correlation and the distinction between occluded or non-occluded regions, resulting in visual artifacts like cluttered textures in the occlusion estimation. Different from all of them, our model adaptively predicts the invisible regions through graph-based region style reasoning. As circled in Figure 6, our model can maintain better shape consistency and generate more reasonable texture for invisible parts. Notably, benefited from our Local Structure Transfer module that can model and transfer local characteristics, our model can better preserve the hat structure as shown in the last row of Figure 6.

Quantitative Comparison
We also compare state-of-the-art methods with our model by numerical cevaluation. Among all the models, our result reaches the highest SSIM and PSNR scores, which indicates that our model maintains the low-level statistical consistency with real images best. We further adopt deep metrics including FID and IS scores to measure the high-level perceptual consistency between our generated images and real images. As shown in Table 1, our model leads the best in FID, which clearly shows the advantages to preserve the texture details. Besides, the IS score of our model surpasses most of prevalent models, which means that our GLocal can generate images with better quality and realism.

Ablation Study
We have trained several variant models to examine the effectiveness of our important components. w/o Focal Loss(w/o FL). This variant predict the segmentation map with the vanilla multi-class cross-entropy loss. w/o Global Graph Reasoning(w/o GGR). This variant removes the graph reasoning block and thus ignoring the distinction between occluded and non-occluded areas. w/o Local Structure Transfer(w/o LST). This variant removes the local structure transfer mechanism and simply takes the features processed by global graph reasoning as the decoder input to generate the final image. GLocal (Ours).This is the full model of our method. From the quantitative result shown in Table 1, we can verify the improvement of our three components. Compared with other variants, our full model outperforms them by a large margin in FID, which indicates that the cooperation of these components can generate more photo-realistic images. Besides, our full model also achieves the leading performance with the best IS, SSIM, and PSNR scores, which means that our full model can improve shape consistency, structure similarity, and pixel-level alignment with the real images. Intuitively, from the visual results in Figure 7, we can see that focal loss helps to achieve more dedicated semantic prediction since the less occupied region cannot be estimated under w/o FL setting (e.g. the segmentation map predicted by w/o FL lacks the sheet region.), which further leads to unrealistic texture rendering. For variants like w/o GGR and w/o LST, it fails to infer the occluded regions and preserve the structure information as depicted by the absurd texture performance, which is circled in Figure 7.

Application
Our GLocal model can also be extended to texture inpainting by inferring the invisible region in a corrupted source image. Given source image whose partial region is masked, the texture inpainting task can reconstruct the source image by filling in the missing region. The corrupted region in texture inpainting can be regarded as the occluded region in pose transfer since they both are invisible in the reference image. Based on this observation, we can utilize the regionwise graph reasoning to reconstruct the original image by calculating the style latent codes of the existing area and propagating the contextual style statistics into the missing regions with graph reasoning. The selection of mask shape divides our application into the following two categories. Irregular region inpainting. The irregular region inpainting produces the inpainted result based on the irregular masked input, whose corrupted mask is provided by Irregular Mask Dataset (Liu et al. 2018). As shown in Figure 8 (a), our method can generate correct structure and consistent textures for irregular regions with visible texture. Semantic region inpainting. For semantic region inpainting, we randomly remove the texture for specific semantic regions (e.g.hair, pants). From the visual results in Figure 8 (b), our approach is capable of recovering the content and rendering reasonable semantics.

Conclusion
In this paper, we have presented a semantic-assisted person image generation framework to synthesis the targetposed source person or inpaint the corrupted image. Our approach models the semantic regions as a human skeletonbased graph and then infers the occluded target region's style with graph reasoning. To transfer the local structure from source to the generation features, the local property is learned with self-correlation and global-varying affine parameters are employed to modulate the generation activations. Extensive experiments and ablation studies have proven the superiority and effectiveness of our model.