{'fraction_non_alphanumeric': 0.03295922184811074, 'fraction_numerical': 0.011447811447811448, 'mean_word_length': 4.814879269088536, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Few-shot Image Generation via Cross-domain Correspondence

Training generative models, such as GANs, on a target domain containing limited examples (e.g., 10) can easily result in overfitting. In this work, we seek to utilize a large source domain for pretraining and transfer the diversity information from source to target. We propose to preserve the relative similarities and differences between instances in the source via a novel cross-domain distance consistency loss. To further reduce overfitting, we present an anchor-based strategy to encourage different levels of realism over different regions in the latent space. With extensive results in both photorealistic and non-photorealistic domains, we demonstrate qualitatively and quantitatively that our few-shot model automatically discovers correspondences between source and target domains and generates more diverse and realistic images than previous methods.


Introduction
Consider 10 portrait paintings by the incomparable Amedeo Modigliani [36], shown in Fig. 1 (middle). Given only these 10 paintings, would it be possible to train a model which can generate infinitely many paintings in the style of Modigliani? Unfortunately, contemporary generative models [11,12,13,29,3] require thousands of images to train properly, not 10. This problem is of practical importance, since many such domains of interest have a very limited collection of images (e.g., there are just 10 examples per artist in the Artistic-Faces dataset [36]).
Transfer learning serves as an alternative to training from scratch and has been explored in the context of generative adversarial networks (GANs) to address the limited data regime. The key idea is to start with a source model, pretrained on a large dataset, and adapt it to a target domain with limited data by either making only small changes to the network parameters to preserve as much information as possible [34,23,32,20,16], or by synthetically increasing the training data via data augmentation [39,11]. Most of these methods, however, are designed for scenarios with more than 100 training images. When the number of available images is lowered to just a few [16], results often overfit to the training samples or are of poor quality.
In this work, we explore transferring a different kind of information from the source domain, namely how images relate to each other, to address the limited data setting. Intuitively, if the model can preserve the relative similarities and differences between instances in the source domain, then it has the chance to inherit the diversity in the source domain while adapting to the target domain. To capture this notion, we introduce a novel cross-domain distance consistency loss, which enforces similarity in the distribution of pairwise distances of generated samples before and after adaptation. Unlike domain adaptation approaches like image-to-image translation, here we are adapting models, not images.
Interesting properties emerge when enforcing this structure-level alignment between the two domains. Specifically, when the source and target domains are related (e.g., faces and caricatures), our approach automatically discovers a one-to-one correspondence between them and is able to more faithfully model the true target distribution in terms of both diversity and image realism, as shown in Fig. 1. When the two domains are unrelated (e.g., cars and caricatures), our approach is unable to model the target distribution but still discovers interesting part-level correspondences to generate diverse samples.
Since the few training samples only form a small subset of the target distribution we seek to approximate, we find it necessary to enforce realism in two different ways, to not inordinately penalize the diversity among the generated images. We apply an image-level adversarial loss on the synthesized images which should map to one of the real samples. For all other synthesized images, we only enforce a patch-level adversarial loss. In this way, only a small subset of our generated samples need to look like one of the few-shot training images, while the rest are only forced to capture their patch-level texture.
Contributions. Our main contribution is a novel GAN adaptation framework, which enforces cross-domain correspondence for few-shot image generation. Through extensive qualitative and quantitative results, we demonstrate that our model automatically discovers correspondences between related source and target domains to generate diverse and realistic images.

Related work
Few-shot learning. Representative few-shot classification approaches include learning a feature similarity function between the query and support examples [30,26] and learning to learn how to adapt a base-learner to a new task [6,22].
Few-shot image generation aims instead to hallucinate new and diverse examples while preventing overfitting to the few training images. Existing work mainly follows an adaptation pipeline, in which a base model is pretrained on a large source domain and then adapted to a smaller target domain. They either embed a small number of new parameters into the source model [23,32] or directly update the source model parameters, using different forms of regularization [20,16]. Others employ data augmentation to reduce overfitting [39,11] but are less effective under the extreme few-shot setting (e.g., 10 images). In contrast to prior work, we regularize the adaptation of the source model by transferring how images relate to each other in the source domain to the target domain, leading to plausible generation results, even with very few examples.
Domain translation. Translating images from the source domain is an alternative approach for generating more target domain data. However, such methods [9,40,41] require a large amount of training data for both source and target domains and are not suitable for the few-shot scenario. Recent work [17,33,25] has begun to address this issue via learning to separate the content and style factor, but requires large amount of labeled data (class or style labels). In our case, we assume access to a large amount of unlabeled data in the source domain and focus on adapting the source model to the target domain for unconditional image generation.
Distance preservation. To alleviate mode collapse in GANs, DistanceGAN [2] proposes to preserve the distances between input pairs in the corresponding generated output pairs. A similar scheme has been employed for both unconditional [27,18] and conditional [19,35] generation tasks to increase diversity in the generations. In our work, we aim to inherit the learned diversity from the source model to the target model and achieve this via our novel cross-domain distance consistency loss.

Approach
We are given a source generator G s , trained on a large source dataset D s , which maps noise vectors z ∼ p z (z) ⊂ Z, drawn from a simple distribution in a low-dimensional space, into images x. We aim to learn an adapted generator G s t by initializing the weights to the source generator and fitting it to a small target dataset D t .
A naive translation can be obtained simply by using a

Source
Overfit → Our → 0 1 2 3 Figure 2: Adapting a source model (FFHQ) to 10 paintings using only the adversarial loss [34] results in overfitting, which leads to a loss in correspondence between source and target images. Our adaptation method preserves this property in a much better way, where the same noise maps to corresponding images in source/target.
GAN training procedure, with a learned discriminator D.
With the non-saturating GAN objective, this corresponds to solving: Previous work [34] shows that this works well when the target dataset size exceeds 1000 training samples. However, in the extreme few-shot setting, this method overfits, as the discriminator can memorize the few examples and force the generator to reproduce them. This is shown in Fig. 2, where we see collapse after tuning the source model (top row) to the few-shot target dataset (middle row).
To prevent overfitting to generate diverse and realistic images (Fig. 2, bottom row), we propose a new crossdomain consistency loss (Sec. 3.1), which actively uses the original source generator to regularize the tuning process, and a "relaxed" discriminator (Sec. 3.2), which encourages different levels of realism over different regions in the latent space. Our approach is shown in Fig. 3.

Cross-domain distance consistency
A consequence of overfitting during adaptation is that relative distances in the source domain are not preserved. As seen in Fig. 2, the visual appearance between z 1 and z 2 collapses, disproportionately relative to z 1 and z 3 , which remain perceptually distinct. We hypothesize that enforcing preservation of relative pairwise distances, before and after adaptation, will help prevent collapse.
To this end, we sample a batch of N + 1 noise vectors {z n } N 0 , and use their pairwise similarities in feature space to construct N -way probability distributions for each image. This is illustrated in Fig. 3 from the viewpoint of z 0 .  1) Cross-domain consistency loss L dist aims to preserve the relative pairwise distances between source and target generations. In this case, the relative similarities between synthesized images from z 0 and other latent codes are encouraged to be similar. (2) Relaxed realism is implemented by using two discriminators, D img for noise sampled from the anchor region (z anch ) and D patch otherwise.

Softmax + KL-Divergence
The probability distribution for the i th noise vector, for the source and adapted generators is given by, where sim denotes the cosine similarity between generator activations at the l th layer. We are inspired by recent methods in contrastive learning [24,4,7], which converts similarities into probability distributions for unsupervised representation learning, as well as perceptual feature losses [10,5,28], which show that activations on multiple layers on discriminative networks help preserve similarity.
We encourage the adapted model to have similar distributions to the source, across layers and image instances by using KL-divergence: (3) While correspondence helps prevent collapse, we also modify the adversarial loss to further prevent overfitting.

Relaxed realism with few examples
With a very small target data size, the definition of what constitutes a "realistic" sample becomes increasingly overconstrained, as the discriminator can simply memorize the few-shot target training set. We note that the few training images only form a small subset of the desired distribution and extend this notion to the latent space. We define "anchor" regions, Z anch ⊂ Z, which form a subset of the entire latent space. When sampled from these regions, we use a full image discriminator D img . Outside of them, we enforce adversarial loss using a patch-level discriminator D patch , To define the anchor space, we select k random points, corresponding to the number of training images, and save them. We sample from these fixed points, with a small added Gaussian noise (σ = .05). We use shared weights between the two discriminators by defining D patch as a subset of the larger D img network [9,40]; using internal activations correspond to patches on the input. The size depends on the network architecture and layer. We read off a set of layers, with effective patch size ranging from 22 × 22 to 61 × 61.

Final Objective
Our final objective consists of just these two terms: L adv for the appearance of the target and L dist , which directly leverages the source model to preserve structural diversity: The patch discriminator gives the generator some additional freedom on the structure of the image. The adapted generator is directly incentivized to borrow the domain structure from the source generator, due to the cross-domain consistency loss. As shown in the top and bottom rows in Fig. 2, the model indeed discovers cross-domain correspondences between source and target domains.
We use the StyleGANv2 architecture 1 [13], pre-trained on a large dataset (e.g. FFHQ [12]) as our source model. We use a batch size of 4. Empirically, we find that a high λ, from 10 3 to 10 4 , to work well. Additional training details can be found in the supplementary.

Experiments
We explore different source → target adaptation settings to analyze the effectiveness of our approach in preserving part-level correspondences between images generated from G s and G s→t . We also investigate what kinds of correspondences emerge when the source and target domains are unrelated.
Baselines: We compare to baselines, which similar to ours, adapt a pre-trained source model to a target domain with limited data. (i) Transferring GANs (TGAN) [34]: finetunes a pre-trained source model to a target domain with the same objective used to train the source model; (ii) Batch Statistics Adaptation (BSA) [23]: only adapts the scale and shift parameters of the model's intermediate layers; (iii) MineGAN [32]: for a given pre-trained source (e.g. MNIST 0-8) and target (e.g. 9) domain, it transforms the original latent space of source to a space more relevant for the target (e.g. mapping all 0-8 regions to 4, being more similar to 9); (iv) Freeze-D [20]: freezes the high resolution discriminator layers during adaptation; (v) Non-leaking data augmentations [11,39]: uses adaptive data augmentations (TGAN + ADA) in a way that does not leak into the generated results; (vi) EWC [16]: extends the idea of Elastic Weight Consolidation [15] for adapting a source model to a target domain, by penalizing large changes to important weights (estimated via Fisher information) in the source model.

Quality and Diversity Evaluation
Qualitative comparison Fig. 4 shows results on two target domains using different methods, all of which start from the same source model pre-trained on FFHQ (Source). We observe that TGAN strongly overfits to the available training data, as was the case for Amedeo paintings (Fig. 2). Using adaptive data augmentations (TGAN + ADA) has little to no effect on sketches, and further degrades the quality for caricatures, where augmentations (e.g. 90 • rotations) leak into the generated images. FreezeD, MineGAN and EWC perform better than TGAN by generating slightly more diverse images. However, the diversity is only introduced through minor modifications among the few captured modes. For example, (i) the caricature results for EWC show multiple generations capturing the same person with different expression/hairlines; (ii) the results on sketches using Mine-GAN depict a person with similar attributes in multiple generations. Our method better captures the distribution of caricatures and sketches, and generates diverse and realistic images containing objects which do not appear in the training images (e.g. hats, in sketches). This is because our method is flexible enough to not penalize the generated images which do not adhere to the 10 training samples.  dataset to train our model. For evaluation purposes, however, we can use the entire dataset to measure how well our generated images model the true distribution. Table 1 shows the FID [8] scores. Our method significantly outperforms all baselines for Babies and Sunglasses. For domains with limited data, however, the FID score would not reflect the overfitting problem.
Ideally, we wish to assess the number of visually distinct images an algorithm can generate. In the worst case, the algorithm will simply overfit to the original k training images. To capture this, we first generate 1000 images and assign them to one of the k training images, by using lowest LPIPS distance [38]. We then compute the average pairwise LPIPS distance within members of the same cluster and then average over the k clusters. A method that reproduces the original images exactly will have a score of zero by this metric. Table 2 summarizes the distances for different baselines over three target domains. We see that

Caricatures
Amedeo's paintings Sketches TGAN [34] 0.39 ± 0.06 0.41 ± 0.03 0.39 ± 0.03 TGAN+ADA [11] 0.50 ± 0.05 0.51 ± 0.04 0.41 ± 0.05 BSA [23] 0. 35   We see that leaving out L dist reduces diversity among the generations, all of which have very similar head structure and hair style. We next study the different ways we enforce realism. What happens if we keep L dist , but use image-level adversarial loss through D img on all generations? 'Ours w/ only D img ' results reveal the problem of mode collapse at the part level (same blue hat appears in multiple generations) and the phenomenon where some results are only slight modifications of the same mode (same girl with and without the blue hat). Could we then only use D patch to enforce patch-level realism on all generations? 'Ours w/ only D patch ' shows the results, where we observe more diversity, but poorer quality compared to 'Ours w/ only D img '. This is because the discriminator never gets to see a complete caricature image, and consequently does not learn the part-level relations which makes a caricature look realistic. 'Ours' combines all the ideas, resulting in generations which are diverse and realistic at both the part and image level.

Analyzing source ↔ target correspondence
As mentioned before, our method captures, to a large extent, the correspondence between the images from source and target domains, and preserves it during the adaptation process. In this section, we study this property in detail in different source to target adaptation settings. Related source/target domains Our method builds around the idea of discovering correspondence between a source and a target domain. But how well does that property actually hold? We first consider the FFHQ → Caricatures/sketches setting in Fig. 4,and analyze the baselines' results. As seen previously in Sec. 4.1, the generated images mostly overfit to the training samples and are unable to borrow anything other than rough pose from the corresponding source. Using our method, on the other hand, we observe that the content of the natural source faces, as well as any of the accessories (e.g. hats/sunglasses), are preserved in both the sketch and caricature domain, depicting a much cleaner correspondence. We further study this for other source→ target settings: (i) Church → Van Gogh houses, (ii) Cars → Abandoned cars, (iii) FFHQ → Sunglasses, as shown in Fig 6. When the source and target domains have similar semantics, the results generated from the same noise vectors in respective domains have clear correspondence.
Unrelated source/target domains We adapt four source models (FFHQ, Church, Cars, and Horses) to two target domains (Caricatures and Haunted houses) and present the results in Fig. 7. For FFHQ → Caricatures and Church → Haunted, the generated results from the adapted model mimic the target domain appearance and reflect correspondences with the source. For all the remaining scenarios, the adaptations do not capture the target distribution accurately. However, some part-level correspondences still 1-shot 5-shot 10-shot Quantitative analysis of source/target relevance We translate four source models to four target domains: caricatures, haunted houses, landscape drawings, and abandoned cars. We then test how well a translated model can embed an unseen image from the respective target domain, E.g., after translating the four source models to the caricature domain, we use a new caricature and embed it into the four models using Image2StyleGAN [1]. The results are shown in Fig. 8. In Table 3, for each domain, we report the average similarity scores between five unseen inputs and their reconstructions. We see that FFHQ, Church, and Cars best reconstruct images from caricatures, haunted houses, and abandoned cars respectively which aligns with our intuition.

Effect of target dataset size
So far, all results are generated with 10 training images per target domain. We now explore how the dataset size affects the quality and diversity of the generated images. We consider two adaptation setups, Church → Landscape drawings and FFHQ → Sketches, and present results in 1-shot, 5-shot and 10-shot settings; see Fig. 9. The real images used in these settings can be found in the supplementary.
In 1-shot, our method introduces small variations to the single target image, for example the lady appears in different poses in the generated sketches, and the mountains and waterfall have different structures. The diversity of the re-

Conclusion and Limitations
We proposed to adapt a pretrained GAN learned on a large source domain to a small target domain by discovering cross-domain correspondences. While our method generates compelling results, it is not without limitations. Cars → Abandoned cars in Fig. 6 depicts an example, where the color of the red car changes to orange in its abandoned form, likely because of the existence of an orange car (and no red one) in the 10 training images. FFHQ → Sunglasses depicts another example, where a blonde hair turns dark with sunglasses. These show that there is a need for discovering better correspondence between the source and target domains, which will lead to more diverse generations. Nevertheless, we believe this work takes an important step towards creating more data-efficient generative models, demonstrating that existing source models can be leveraged in an effective way to model new distributions with less data. caricature domain best embeds and reconstructs unseen images, indicating that caricature as a domain is most related to FFHQ than any other source domain. Fig. 13 presents an extension of Fig. 9 of main paper, where we study the 1shot, 5-shot and 10-shot setting for two baselines, and compare it with our method. We notice that both FreezeD and EWC, overfit to the target sample in 1-shot setting, generating virtually identical sketches/scenes. This trend of overfitting for these baselines continues in 5/10-shot settings as well, where generations collapse to small variations around a few modes. Our method, on the other hand, takes the benefit of increasing training data size, by learning to generate more and more diverse samples, different from the images used for training.
Visualizing the clusters Fig. 14 visualizes the clusteringbased diversity assessment introduced in Sec. 4.1 of the main paper. We group the generated images from a method into k clusters, with the k training images serving as the cluster center. After this, we study the resulting clusters, where we visualize how similar is (i) the closest member to the center (measured via LPIPS), (ii) the farthest member to the center. The intuition is that a method whose generated images overfit to the training data will result in clusters where the closest member is very similar to the corresponding cluster center. Each column deals with one cluster, where the cluster centers (real images used for training) are shown in the middle. The top half of the figure visualizes the closest members for different methods, whereas the bottom half visualizes the farthest ones. When no images get assigned to a cluster, the concept of closest/farthest members doesn't apply, and we depict this with a red cross. Summarily, we observe that the closest members from TGAN/EWC are much more similar to the corresponding center than our method, whose even closest members are visually distinct. This observation also helps explain the better performance of our method compared to others in Table 2 (main paper).
Hand gestures experiment We find the property of emerging correspondences within seemingly unrelated source/target domains interesting, and hence for creativity purposes, take a further step to explore the idea. We collect images of arbitrary hand gestures being performed over a plain surface, and train a source model from scratch using that dataset. Next, we adapt it to various domains such as landscapes, fire, maps. During inference, we observe different aspects of the target domains a pair of hands can control (e.g. structure of river/islands). Please see our teaser video, which shows the correspondence results in this case, as well as better explains the benefits of our method in previously discussed scenarios (e.g. FFHQ → caricatures, Church → Van Gogh houses).
Precision and recall metrics A limitation of FID [8] is that it packs two aspects of the generated images, sample quality and diversity, into one score. This makes it difficult to disentangle and study the two properties separately.
To overcome this, density and coverage metrics were proposed to evaluate the generative models [21]. In some feature space (e.g. CNN embeddings), density measures how many real-sample neighbourhood regions contain a fake sample. Coverage, in the same space, measures the ratio of real samples whose neighbourhood contains at least one fake sample. In both the definitions, neighbourhood is defined as a spherical region around a real sample, with its radius given by the distance from the next nearest real sample. A high score for both the metrics is preferred. Density is unbounded, whereas coverage is bounded by 1. We present evaluation of the baselines fare using these metrics on FFHQ babies dataset in Table 4. We observe that Mine-GAN achieves a superior density score, i.e. quality of the generated image, but suffers in the coverage aspect. This is again an indication of mode collapse to a small number of high quality samples. Our method achieves a better balance between the quality as well as diversity of the generated samples. Note that this result is in alignment with the one presented in Table 2 (main paper), which studies diversity among the generated samples in a different way.

Haunted Houses
Landscape drawings

Raphael paintings
Otto Dix's paintings Maps Figure 10: Real images used for translating a source model to different target domains. For landscape drawings and sketches, we have shown the images used in 1-shot and 5-shot scenario, for the experiment presented in Fig. 9 of the main paper, and  Figure 11: Translating the FFHQ source model to different target domains. The noise vector is kept same across the columns, so that we can study the relation between the corresponding source and target image.  Figure 13: Comparison of our method compared to EWC [16] and FreezeD [20] in 1-shot, 5-shot and 10-shot setting.  Figure 14: Visualizing the clusters formed using the technique described in Sec. 4.1 of the main paper. The closest members produced by TGAN/EWC are much more similar to the corresponding cluster center than our method, indicating that the generations using the proposed method possess more diversity.