{'fraction_non_alphanumeric': 0.03938449467099671, 'fraction_numerical': 0.01992110023757111, 'mean_word_length': 4.7953770367563475, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

Generative adversarial networks (GANs) typically require ample data for training in order to synthesize high-fidelity images. Recent studies have shown that training GANs with limited data remains formidable due to discriminator overfitting, the underlying cause that impedes the generator's convergence. This paper introduces a novel strategy called Adaptive Pseudo Augmentation (APA) to encourage healthy competition between the generator and the discriminator. As an alternative method to existing approaches that rely on standard data augmentations or model regularization, APA alleviates overfitting by employing the generator itself to augment the real data distribution with generated images, which deceives the discriminator adaptively. Extensive experiments demonstrate the effectiveness of APA in improving synthesis quality in the low-data regime. We provide a theoretical analysis to examine the convergence and rationality of our new training strategy. APA is simple and effective. It can be added seamlessly to powerful contemporary GANs, such as StyleGAN2, with negligible computational cost.


Introduction
While state-of-the-art GANs like StyleGAN2 [21] are constantly pushing forward the fidelity and resolution of synthesized images, they usually require a large amount of training data to fully unleash their power. Training GANs with insufficient data tend to generate poor-quality images, as shown in Figure 1. Collecting sufficient data samples for these GANs is sometimes infeasible, especially in domains where data are sparse and privacy-sensitive. To ease the practical deployment of powerful GANs, it is necessary to devise new strategies for training GANs with limited data while preserving the quality of synthesis.
Recent studies have shown that the overfitting of the discriminator is the critical reason that impedes effective GAN training on limited data [3,19,55,46], rendering severe instability of training dynamics. Specifically, when the discriminator starts to overfit, the distributions of its outputs for real and generated samples gradually diverge from each other [19,46], and its feedback to the generator becomes less informative. Consequently, the generator converges to an inferior point, compromising the quality of synthesized images. Recent solutions to this problem include the use of standard data augmentations, either conventional or differentiable, to real and generated images [19,55,57,45] or applying an additional model regularization term [46]. Addressing the discriminator overfitting is still an open problem. We are interested in finding an alternative way to the aforementioned approaches.
generator in a GAN itself to provide the augmentation, a more natural way to regularize the overfitting of the discriminator. Compared to the model regularization, our approach is more adaptive to fit different settings and training status without manual tuning. Specifically, APA takes the fake/pseudo samples synthesized by the generator and moderately feeds them into the limited real data. Such pseudo data are adaptively presented to the discriminator as "real" instances. The goal of this pseudo augmentation for the real data is not to enlarge the real dataset but to suppress the discriminator's confidence in distinguishing real and fake distributions. The deceit is introduced adaptively, which is moderated by a deception probability according to the degree of overfitting. To quantify overfitting, we study a series of plausible heuristics derived from the discriminator raw output logits.
The main contribution of this work is a novel adaptive pseudo augmentation method for training GANs with limited data. This approach deceives the discriminator adaptively and mitigates the problem of discriminator overfitting. The proposed APA can be readily added to existing GAN training with negligible computational cost. We conduct extensive experiments to demonstrate the effectiveness of APA for state-of-the-art GAN training with limited data. The results are comparable or even better than other types of solutions [19,46]. APA is also complementary to existing methods based on standard data augmentations for gaining a further performance boost. Besides, we theoretically connect APA with minimizing the JS divergence [27] between the smoothed data distribution and generated distribution, proving its convergence and rationality. We hope that our approach could extend the breadth and potential of solutions to GAN training with limited data.

Related Work
Generative adversarial networks. Generative adversarial networks (GANs) [10,32,39,52] adopt an adversarial training scheme, where a generator keeps refining its capability in synthesizing images to compete with a discriminator (i.e., a binary classifier) until the discriminator fails to classify the generated samples as fakes. GANs are known to suffer from training instability [10,42,3,31]. Various approaches have been proposed to stabilize the training and improve the quality of synthesis by minimizing different f -divergences of the real and fake distributions [34]. The saturated form of vanilla GAN [10] is theoretically proven to minimize the JS divergence [27] between the two distributions. LSGAN [30] and EBGAN [54] correspond to the optimizations of χ 2 -divergence [38] and the total variation [4], respectively. On another note, WGAN [4] is designed for minimizing the Wasserstein distance.
Training GANs with limited data. The significance and difficulty of training GANs with limited data have been attracting attention from many researchers recently. The issue of data insufficiency tends to cause overfitting in the discriminator [49,19,55], which in turn deteriorates the stability of training dynamics in GANs, compromising the quality of generated images.
Many recent studies [53,56,45,57,55,19] propose to apply standard data augmentations for GAN training to enrich the diversity of the dataset to mitigate the overfitting of the discriminator. For instance, DiffAugment (DA) [55] adopts the same differentiable augmentation to both real and fake images for the generator and the discriminator without manipulating the target distribution. Adaptive discriminator augmentation (ADA) [19] shares a similar idea with DA, while it further devises an adaptive approach that controls the strength of data augmentations adaptively. In this work, we extend the study of such an adaptive approach.
Another type of solution is model regularization. Previous efforts on regularizing GANs include adding noise to the inputs of the discriminator [3,44,15], gradient penalty [31,11,41], one-sided label smoothing [42], spectral normalization [33], label noise [7], etc. These methods are designed for stabilizing training or preventing mode collapse [42]. The essence of their goals could be considered similar to our method since training GANs in the low-data regime exhibits similar behaviors as previously observed in early GANs with sufficient data. Additional discussions on these techniques are provided in the Appendix. Under the limited data setting, a very recent study proposes an LCregularization term [46] to regulate the discriminator predictions using two exponential moving average variables that track the discriminator outputs throughout training.
Our work explores an alternative solution from a different perspective, which is also complementary to previous approaches based on standard data augmentations.

Methodology
In GAN's adversarial training, the goal of the generator G is to deceive the discriminator D and maximize the probability that D makes a wrong judgment. Therefore, G keeps refining its generated samples to better deceive D over time. When the training only accesses a limited amount of data, one would observe that D turns out to be overly confident and hardly makes any mistake, causing its feedback to G to become meaningless.
In Figure 2, we show the training "snapshots" of two StyleGAN2 [21] models on the FFHQ dataset [20]. The settings of the two models differ only by the amount of data available to them for overfitting heuristic Figure 3: Adaptive pseudo augmentation (APA) for GAN training with limited data. We employ a GAN to augment itself using the generated images to deceive the discriminator adaptively. Specifically, APA feeds the images synthesized by the generator into the limited real data moderately, and these fakes are presented as "real" instances to the discriminator. Such deceits are introduced adaptively using an overfitting heuristic λ defined by the discriminator raw output logits. The augmentation/deception probability p can be adaptively controlled throughout training.
training. As can be observed, both training processes start smoothly, and distributions of discriminator outputs for the real and generated images overlap at the early stage. As the training progresses, the discriminator, which only has access to limited data (7k images, 10% of full data), experiences diverged predictions much more rapidly, and the average sign boundary turns out to be more apparent. This divergence in prediction shows that D becomes increasingly confident in classifying real and fake samples. At the late stage of training, D can even judge all the input samples correctly with high confidence. Meanwhile, the evaluation FID [12] scores (lower is better) deteriorate, consistent with the divergence of D's predictions. The phenomena above demonstrate how a discriminator gets overfitted quickly with limited data. As can be seen from the FID curves, the overfitting directly influences training dynamics and the convergence of G, leading to poor generation performance.

Adaptive Pseudo Augmentation
The generator itself naturally possesses the capability to counteract discriminator overfitting. To harness this capability, our method employs a GAN to augment itself using the generated samples (see Figure 3). Specifically, we feed the samples generated by G into the limited real data to form a pseudo-real set. The fake images in this set will be adaptively presented to D, pretending themselves as real data. The goal here is to deceive D with the pseudo-real set and consequently suppressing its confidence in distinguishing real and fake images.
Blindly presenting the fake images as reals to D may weaken the fundamental ability of D in adversarial training. In our approach, the deceit is introduced adaptively to avoid any potential adverse effects. To moderate the deception, we perform pseudo augmentation based on a probability, p ∈ [0, 1), that quantifies the deception strength. Specifically, the pseudo augmentation will be applied with the probability p or be skipped with the probability 1 − p.
We note that the overfitting state of D is dynamic throughout the training (see Figure 2). It is intuitive to let the deception probability p be adjusted adaptively based on the degree of overfitting. Ideally, p should be adjusted without manual tuning regardless of data scales and properties. To achieve this goal, inspired by ADA [19], we apply an overfitting heuristic λ that quantifies the degree of D's overfitting. We extend the control scheme of ADA and provide three plausible variants: where D real and D fake are defined as D real = logit (D (x)) , D fake = logit (D (G (z))) , where logit denotes the logit function. As shown in Figure 2, the λ r in Eq. (1) estimates the portion of real images that obtain positive logit predictions by D, and that of generated images is captured by λ f . Besides, λ rf indicates half of the distance between the signs of the real and fake logits. For all these heuristics, λ = 0 represents no overfitting, and λ = 1 means complete overfitting. We use λ r in our main experiments and study other variants in the ablation study.
The strategy of using λ to adjust p is as follows. We set a threshold t (in most cases of our experiments, t = 0.6) and initialize p to be zero. If λ signifies too much/little overfitting regarding t (i.e., larger/smaller than t), the probability p will be increased/decreased by one fixed step. Using this step size, p can increase from zero to one in 500k images shown to D. We adjust p once every four iterations and clamp p from below to zero after each adjustment. In this way, the strength of pseudo augmentation can be adaptively controlled based on the degree of overfitting (see Figure 3).

Theoretical Analysis
Let p z (z) be the prior on the input noise variable. The mapping from the latent space to the image space is denoted as G (z). For sample x, D (x) represents the estimated probability of x coming from the real data. To examine the rationality of APA, we analyze it in a non-parametric setting [10], where a model is represented with infinite capacity by exploring its convergence in the space of probability density functions. Ideally, the estimated probability distribution p g defined by G should perfectly model the real data distribution p data without bias if given enough capability and training time.
Since the deception strength p is adaptively adjusted, to facilitate this theoretical analysis, we make a mild assumption that α is the expected strength that approximates the effect of dynamic adjustment during the entire training procedure. Since p ∈ [0, 1), we have 0 ≤ α < p max < 1, where p max is the maximum of deception strength throughout training. Hence, the value function V (G, D) for the minimax two-player game of APA can be reformulated as: First, let us consider the optimal discriminator [10] for any given generator.

Proposition 1.
If the generator G is fixed, the optimal discriminator D for APA is: Proof. Applying APA, given any generator G, the training objective of the discriminator D is to maximize the value function V (G, D) in Eq. (3): For any (m, n) ∈ R 2 \ {0, 0}, the function f (y) = m log (y) + n log (1 − y) achieves its maximum in the range [0, 1] at m m+n . Besides, the discriminator D is defined only inside of supp (p data ) ∪ supp (p g ), where supp is the set-theoretic support. Therefore, we conclude the proof for Proposition 1.
We have got the optimal discriminator D * G (x) in Eq. (4) that maximizes the value function V (G, D) given any fixed generator G. The goal of generator G in the adversarial training is to minimize the value function V (G, D) in Eq. (3) when D achieves the optimum. Since the training objective of D can be interpreted as maximizing the log-likelihood for the conditional probability P (Y = y|x), where Y estimates that x comes from p data (i.e., y = 1) or from p g (i.e., y = 0), we reformulate virtual training criterion [10] as: Then, let us consider the global minimum of C (G) trained with the proposed APA.
Proposition 2. Applying APA, the global minimum of the virtual training criterion C (G) is still achieved if and only if p g = p data , where C (G) = − log 4.
is the global minimum of C (G), and it can only be achieved when p g = p data , as in the derivation of Eq. (5), we obtain: Subtracting Eq. (8) from Eq. (7), By substituting Eq. (4) into Eq. (9), we achieve: where KLD is the Kullback-Leibler (KL) divergence. Moreover, Eq. (10) further implies that the generation process of G by APA can be regarded as minimizing the Jensen-Shannon (JS) divergence between the smoothed data distribution and the generated distribution: For the two distributions P and Q, their JS divergence JSD (P Q) ≥ 0 and JSD (P Q) = 0 if and only if P = Q. Therefore, for 0 ≤ α < p max < 1, we obtain that C * (G) = − log 4 is the global minimum of C (G), and the only solution is (1 − α) p data + α p g = p g , i.e., p g = p data . Q.E.D.
Given the proof in [10], if G and D have enough capacity to reach their optimum, Proposition 2 indicates that the generated distribution p g can ideally converge to the real data distribution p data . So far, we have proved the convergence of G trained with our proposed APA, which can perfectly model the real data distribution given sufficient capability and training time. Besides, the JS divergence term between the smoothed data distribution and the generated distribution in Eq. (11) implies that the judgment of D may be moderated to alleviate overfitting. These conclusions explain the rationality of the proposed APA for training GANs with limited data.
We exploit some of their artificially limited subsets under different settings. All the images are resized to a moderate resolution of 256 × 256 using a high-quality Lanczos filter [25] to reduce the energy consumption for large-scale GAN training while preserving image quality. Additional dataset details, including the data source and license information, can be found in the Appendix.
Evaluation metrics. We follow the standard evaluation protocol [6,46] for the quantitative evaluation. Specifically, we use the Fréchet Inception Distance (FID, lower is better) [12], which quantifies the distance between distributions for the real and generated images. FID evaluates the realness of synthesized images. Following [12,19], we calculate FID for the models trained with limited
Implementation details. We choose the state-of-the-art StyleGAN2 [21] as the backbone to verify the effectiveness of APA on limited data. We use the default setups of APA provided in Section 3.1 unless specified otherwise. For a fair and controllable comparison, we reimplement all baselines and run the experiments from scratch using official code. All the models are trained on 8 NVIDIA Tesla V100 GPUs. Please refer to the Appendix for more implementation details and additional benchmark results (e.g., performance on BigGAN [6]).

The Effectiveness of APA
Effectiveness on various datasets. The comparative results of StyleGAN2 on various datasets with limited data amounts are shown in Figure 4. The quality of images synthesized by StyleGAN2 deteriorates under limited data. Ripple artifacts appear on the cat faces and human faces, and the facial features of the anime faces are misplaced. On the bird dataset with heavy background clutter, the generated images are completely distorted albeit trained with more data. The proposed APA significantly ameliorates image quality on all these datasets, producing much more photorealistic results. The quantitative evaluation results are reported in Table 1. Applying APA contributes to a performance boost of FID and IS in all cases, suggesting that the synthesized images are with higher quality and diversity on different datasets.
Effectiveness given different data amounts. The comparative results on subsets of FFHQ [20], with varying amounts of data, are shown in Figure 5. The corresponding quantitative test results are StyleGAN2

APA (Ours)
7k (10% data) 70k (100% data) 5k (~7% data) 1k (~1.4% data) Figure 5: The effectiveness of APA to improve StyleGAN2 [21] synthesized results (256 × 256, no truncation) on the subsets of FFHQ [20] with different data amounts. Table 2: The FID (lower is better) and IS (higher is better) scores (256 × 256) of our method on StyleGAN2 trained using the subsets of FFHQ [20] with different data amounts. presented in Table 2. APA improves the image quality and metric performance in all cases. Notably, the quality of synthesized images by APA on 5k/7k data is visually close to StyleGAN2 results on the full dataset while with an order of magnitude fewer training samples. As for the quantitative results, the IS score of APA on 1k data is better than that of StyleGAN2 on 5k data, and both metrics of APA on 5k data outperform StyleGAN2 results on 7k data. APA can even improve StyleGAN2 performance on the full dataset, further indicating its potential.
Overfitting and convergence analysis. As shown in Figure 6, the divergence of StyleGAN2 discriminator predictions can be effectively restricted on FFHQ-7k (10% of full data) by applying the proposed APA. The curves of APA on FFHQ-7k become closer to that of StyleGAN2 on FFHQ-70k, suggesting the effectiveness of APA in curbing the overfitting of the discriminator. Besides, APA improves the training convergence of StyleGAN2 on limited data, shown by the FID curves. More overfitting and convergence analysis can be found in the Appendix.

Comparison with Other Solutions for GAN Training with Limited Data
Performance and compatibility. We compare the proposed APA with representative approaches designed for the low-data regime, including ADA [19] and LC-regularization (LC-Reg) [46], which perform standard data augmentations and model regularization, respectively. The results are reported in Table 3. As a single method, APA outperforms previous solutions in most cases, effectively   improving the StyleGAN2 baseline on both the limited and full datasets. Although ADA [19] achieves slightly better results than our method on FFHQ-5k, it yields a worse FID score with StyleGAN2 on the full dataset. Applying LC-Reg [46] needs careful manual tuning, and its own effect on limited data is not apparent compared to other methods.
It is noteworthy that APA is also complementary to existing methods based on standard data augmentations, e.g., ADA [19]. As can be observed in Table 3, APA can further boost the performance of StyleGAN2 with ADA [19] given limited training data, suggesting the compatibility of our approach with standard data augmentations. Combining ADA [19] and APA on FFHQ full data outperforms StyleGAN2 but is slightly inferior to applying APA solely. The degraded performance is mainly affected by ADA [19], which we empirically found might slightly harm the performance when the training data is sufficient. Overall, these methods are not dedicated to improving performance under sufficient data. Nevertheless, as a beneficial side effect, the proposed APA may have this potential.
More comparative results are included in the Appendix.
Training cost. We compare the computational cost of APA against ADA [19], using the same basic official codebase. There is no parameter or memory increment for both methods. As for the time consumption, we test the training cost on 8 NVIDIA Tesla V100 GPUs. On FFHQ-5k (256 × 256), the average training time of the StyleGAN2 [21] baseline is (4.740 ± 0.100) sec/kimg (i.e., seconds per thousand of images shown to the discriminator). The cost of our method is negligible, slightly increasing this value to (4.789 ± 0.078) sec/kimg. As a reference, the value for ADA [19] is (5.327 ± 0.116) sec/kimg, spending additional time for applying external augmentations.

Comparison with Previous Techniques for Regularizing GANs
As mentioned in Section 2, APA is closely related to previous techniques for regularizing GANs. The comparative results on APA and some representative conventional techniques, i.e., instance noise [44] and one-sided label smoothing (LS) [42], are shown in Table 4. Applying instance noise [44] may not boost the performance of StyleGAN2 much under limited data. One-sided label smoothing (LS) [42] (with the real label of 0.9) outperforms StyleGAN2 but still has a huge performance gap with our method. This further suggests the effectiveness and usefulness of APA.

Ablation Studies
Ablation studies on variants of APA. We study three key elements of APA, i.e., the overfitting heuristic λ, the deception strength p, and the deception strategy. The version used in our main experiments is denoted as the "main" version, where λ = λ r , and p is adjusted adaptively. Besides, the "main" version applies the deception strategy that is analogous to one-sided label flipping. As reported in Table 5, when using other variants of λ we suggested in Eq. (1) (i.e., λ = λ f and λ = λ rf ), the models achieve comparable performance as the main version. The FID score becomes even better for λ = λ rf , indicating the flexibility of our provided heuristics to extend and modify APA. More interestingly, even a fixed moderate deception probability (e.g., p = 0.5) can still work much better Table 5: Ablation studies on variants of APA on FFHQ-5k [20] (256 × 256). We study three key elements of APA, i.e., the overfitting heuristic λ, the deception strength p, and the deception strategy. The "main" denotes the main version used in our previous experiments (i.e., λ = λ r , p is adaptively adjusted, and the deception strategy is analogous to one-sided label flipping).  than original StyleGAN2 on limited data, albeit slightly inferior to the adaptively adjusted p. This implies the importance of the pseudo augmentation, and the adaptive control scheme can further boost performance without manual tuning. As for the deception strategy, we empirically observe that two-sided label flipping can still outperform StyleGAN2 but is inferior to the main version.
Ablation studies on the threshold t. We further provide the ablation studies on the threshold value t in Table 6. The version used in our main experiments is denoted as the "main" version, where t = 0.6. It can be seen that the models with different values of t achieve comparable results, outperforming the StyleGAN2 baseline. On FFHQ-5k (256 × 256), t = 0.6 could be a more plausible choice. For a further explanation, we use t = 0.6 as the default value since it works well in most cases. In practice, the value of t could be further adjusted to achieve even better results. Empirically, a smaller t can be chosen when one has fewer data. This means the deception strength p can be adjusted to increase more rapidly since the discriminator is more prone to overfitting when the data amount is fewer.

Discussion
We have shown the effectiveness of the proposed adaptive pseudo augmentation (APA) for stateof-the-art GAN training with limited data empirically. With negligible computational cost, APA achieves comparable or even better performance than other types of solutions on various datasets. APA is also complementary to existing methods based on standard data augmentations.
Limitations. Despite promising results, the quality of synthesized images by APA on the datasets with extremely limited data amount (e.g., hundreds of images) can still be improved. Besides, on certain datasets such as FFHQ-5k, applying APA solely may be slightly inferior to approaches based on standard data augmentations. Since we do not apply any external augmentations, these two limitations are both due to the insufficiency of the dataset's intrinsic diversity. These limitations may be approached in the future in two ways: 1) Incorporating better standard data augmentations to APA. 2) Exploring the issue of data insufficiency from the generator aspect, e.g., using a multi-modal generator [43] to enhance diversity. In addition, we only theoretically verified the convergence and rationality of APA. In future work, the theoretical analysis on the effectiveness of APA could be further explored.
Broader impact. On the one hand, the effectiveness of APA with negligible computational cost will benefit the practical deployment of GANs, especially in the low-data regime. APA may also extend the breadth and potential of solutions to training GANs with limited data and benefit downstream tasks, such as conditional synthesis. On the other hand, APA may also bring potential concerns on its capability to ease the higher-quality fake media synthesis using only limited data, as technology is usually a double-edged sword. However, we believe that these concerns can be resolved by developing better media forensics methods and datasets as countermeasures.
Let the virtual training criterion [10] for the generator G be C (G). The global minimum of C (G) is achieved if and only if p g = p data , and the minimum value is − log 4, as proved by [10]. This indicates that GANs can perfectly model the real data distribution if given sufficient capacity and time. In practice, we usually use a non-saturated form for G and train it to maximize log D (G (z)) instead of minimizing log (1 − D (G (z))) to ensure a healthy gradient at the early training stage.
In all the figures of this work, we show raw output logits of D before the last Sigmoid activation to better visualize its prediction confidence. Let logit denotes the logit function, we define:

B Additional Discussions on Previous Techniques for Regularizing GANs
As mentioned in our main paper, previous techniques for regularizing GANs include adding noise to the inputs of the discriminator [3,44,15], gradient penalty [31,11,41], one-sided label smoothing [42], spectral normalization [33], label noise [7], etc. These approaches are designed for stabilizing training or preventing mode collapse [42]. The essence of their objectives could be considered similar to our method since training GANs in the low-data regime exhibits similar behaviors as previously observed in early GANs with sufficient data. However, several differences are worth highlighting.
First of all, the actual goals of previous strategies and the proposed APA may not be completely the same. Specifically, APA is specialized for training GANs in the low-data regime, which was not carefully considered by prior studies. This difference in the data setting is quite important since it is the core problem we wish to address. Even the performance of state-of-the-art StyleGAN2 [21] deteriorates when trained with a limited amount of data, although it has exploited many advanced techniques for stabilizing training or preventing mode collapse, such as R1 regularization [31]. The main challenge that lies within the low-data regime is the overfitting of the discriminator. Although this issue might also appear on early GANs, it becomes more severe when data is limited.
Besides, we have presented the comparative studies on APA and conventional techniques in the main paper. Empirically, we showed that previous methods could not boost the performance of StyleGAN2 much under limited data. Some of them showed a huge performance gap in comparison to the proposed APA. The experiments indicate that previous relevant strategies cannot handle the low-data regime well, further suggesting the effectiveness and usefulness of APA.
Last but not least, APA and these techniques themselves are not exactly the same. The proposed APA is an effective practice and improvement of these ideas on modern GANs, whose implementations are very different from the early ones. Compared to previous techniques, APA is more adaptive to fit different settings and the overfitting status in training. Although using adaptive heuristics was also explored in the past, it had been found unpractical at the time [7]. APA makes the adaptive control scheme possible in practice.
We believe that the proposed APA could contribute to the community for its effectiveness, simplicity, and adaptability for training state-of-the-art GANs in the low-data regime. Hopefully, our approach could extend the breadth and potential of solutions to GAN training with limited data.

C Dataset Details
This section will detail our explored datasets in the main paper. We randomly select subsets to confine the size of large datasets and directly use small datasets for GAN training with limited data under different settings. Our main paper focuses on the fundamental unconditional image synthesis task with powerful contemporary GANs. Thus, there is no need to split a separate test set. We exploit a high-quality Lanczos filter [25] for image resizing and save images in the uncompressed PNG format. We will detail each dataset separately as follows.
• AFHQ-Cat. We use the AFHQ-Cat dataset released by [9], which is the cat category of the high-quality Animal Faces-HQ (AFHQ) dataset [9]. The original authors mentioned that they collected images with permissive licenses from the Flickr and Pixabay websites. All images were vertically and horizontally aligned at the center. The dataset was released under Creative Commons BY-NC 4.0 license by NAVER Corporation. As a small dataset, we exploit 5, 153 training images of cat faces with various breeds. The original resolution is 512 × 512, which is scaled to 256 × 256 for training.
• FFHQ. The full Flickr-Faces-HQ (FFHQ) dataset [20] consists of 70, 000 high-quality (1024 × 1024) human face images. The faces contain considerable variation in terms of age, ethnicity, background, and accessories. The original authors mentioned that the images were crawled from the Flickr website, and only the ones under permissive licenses were collected. The images were automatically aligned [22] and cropped. The individual images were published in Flickr by their respective authors under either Creative Commons BY 2.0, Creative Commons BY-NC 2.0, Public Domain Mark 1.0, Public Domain CC0 1.0, or U.S. Government Works license. The dataset itself was made available under Creative Commons BY-NC-SA 4.0 license by NVIDIA Corporation. We randomly select different subsets of FFHQ, i.e., FFHQ-1k (1, 000 images, ∼ 1.4% data), FFHQ-5k (5, 000 images, ∼ 7% data), FFHQ-7k (7, 000 images, 10% data), and FFHQ-70k (70, 000 images, 100% data) to perform GAN training given different data amounts. The original images are resized to 256 × 256 for training in the experiments of our main paper. We will also show some synthesized examples with the original resolution in this Appendix.
• Danbooru2019 Portraits (Anime). The Danbooru2019 Portraits (Anime) [5] is a dataset consisting of 512 × 512 anime faces cropped from solo SFW Danbooru2019 images [2]. The full dataset contains a total of 302, 652 images in a broad portrait style encompassing ears, necklines, hats, etc., rather than the tightly cropped faces. The dataset was released under the Creative Commons public domain (CC-0) license. We artificially confine the dataset into a subset for GAN training, i.e., Anime-5k (5, 000 images, ∼ 2% data). The training image size is 256 × 256.
• Caltech-UCSD Birds-200-2011 (CUB). The Caltech-UCSD Birds-200-2011 (CUB) [47] is an extended version of CUB-200 [50], a challenging dataset containing 200 bird species. The dataset consists of 11, 788 bird images at diverse locations with heavy background clutter. The images were harvested using the Flickr image search. We were unable to find the license information about this dataset. We employ all the available images for GAN training since the dataset is already small and difficult. The resolutions of original images vary, and we uniformly resize them to 256 × 256.

D Implementation Details
In the main paper, we choose the state-of-the-art StyleGAN2 [21] as the backbone to verify the effectiveness of the proposed APA on limited data. Besides, we compare our method with representative approaches designed for the low-data regime, including the adaptive discriminator augmentation (ADA) [19] and LC-regularization (LC-Reg) [46], which perform standard data augmentations and model regularization, respectively. In addition, we compare APA with representative conventional techniques for regularizing GANs, i.e., instance noise [44] and one-sided label smoothing [42]. For a fair and controllable comparison, we reimplement all baselines and run the experiments from scratch using official code. The qualitative and quantitative results of each method are reported using the best model throughout training.
We ran our experiments on an internal computing cluster with Slurm Workload Manager. All the models are trained on 8 NVIDIA Tesla V100 GPUs with 32 GB memory capacity. We follow [19] and employ its mixed-precision FP16/FP32 training scheme in all our experiments. The actual memory consumption for each model is around 11 GB per GPU. We perform 25, 000 kimg (i.e., thousands of images shown to the discriminator, measuring the training progress [20,21,19]) of training for each model. For the average training time cost of different models, please refer to the Training cost of Section 4.2 in the main paper.
We implemented the proposed APA on top of the official implementation of StyleGAN2 [21]. The network architecture is kept unchanged. The mapping network contains 8 fully connected layers, and the dimensionality of the input and intermediate latent space is 512. We use the combination of a generator with output skips and a residual discriminator. The detailed structures of the generator and the discriminator are the same as [21], e.g., using weight demodulation [21] in the generator. The activation function is Leaky ReLU with a negative slope of 0.2. We apply several other standard techniques in [18,20,21], including the mini-batch standard deviation layer at the end of the  discriminator [18], equalized learning rate for all the trainable parameters [18], pixel-wise feature vector normalization [18], the exponential moving average of generator weights [18], style mixing regularization [20], path length regularization [21], and lazy regularization [21]. The training loss is the non-saturating logistic loss [10,21] with R 1 regularization [31]. The batch size is 64 for our experiments trained with the resolution of 256 × 256. The Adam [23] optimizer is applied with β 1 = 0, β 2 = 0.99. For other network and training details, we follow the original paper and official code of StyleGAN2 [21].
For APA, we set the overfitting heuristic λ = λ r in our main experiments and study other variants (i.e., λ = λ f and λ = λ rf ) through the ablation study. Aside from the ablation study of a fixed deception probability p = 0.5, p is adaptively adjusted according to λ. The adaptive adjustment of p is as follows. We first initialize p to be zero and set a threshold value t (t = 0.6 in most cases unless specified otherwise). If λ signifies too much/little overfitting regarding t (i.e., larger/smaller than t), the probability p will be increased/decreased by one fixed step. Using this step size, p can increase quickly from zero to one, i.e., in 500k images shown to D. We adjust p once every four iterations. We clamp p from below to zero after each adjustment so that p can always be larger than zero. We do not set an upper bound for p while it can be naturally restricted under a safe limit. Then, the pseudo augmentation of each instance will be applied with the probability p or be skipped with the probability 1 − p. In this way, the strength of pseudo augmentation can be adaptively controlled based on the degree of overfitting throughout training.
As for other methods used for comparison, we strictly follow all the details in their papers and released official code using recommended setups.

E More Results and Analysis
Effectiveness of APA for transfer learning. Table 7 reports the additional transfer learning results on MetFaces [19] from the pre-trained StyleGAN2 model on FFHQ-70k [20]. All the metrics can be boosted by APA, further verifying its effectiveness for transfer learning besides training from scratch.
The performance of APA under a lower amount of data (i.e., fewer than 1k). We have reported transfer learning results on MetFaces-500 [19] (a subset of 500 images, ∼ 37% of full data) in Table 7. Table 8 shows more results on FFHQ-500 [20] (a subset of 500 images, ∼ 0.7% of full data). Under fewer data, APA can still boost StyleGAN2 performance by a large margin. However, the quality itself of synthesized images by APA can still be improved, consistent with our discussion on limitations in the main paper.
More examples of the effectiveness of APA on various datasets. We show more comparative results of StyleGAN2 [21] and the proposed APA to demonstrate the effectiveness of our method to improve the state-of-the-art baseline on various datasets with limited data amounts, i.e., AFHQ-Cat-5k [9] (Figure 7), FFHQ-5k [20] (Figure 8), Anime-5k [5] (Figure 9), and CUB-12k [47] (Figure 10). Regardless of applying the truncation trick [6,20,21] or not, the proposed APA can significantly ameliorate the degraded synthesis quality of StyleGAN2 with limited training data in all cases. The generated images by our method are highly photorealistic using only limited training data, being closer to the real data distributions.
Qualitative results of comparison with other solutions for GAN training with limited data. In the main paper, we have quantitatively shown the effectiveness of the proposed APA over other state-

APA (Ours)
AFHQ-Cat-5k (5,153 img, 100% data) FFHQ-5k (5,000 img, ~7% data) LC-Reg ADA ADA + APA (Ours) Figure 11: The synthesized results (256 × 256, no truncation) of our method compared to other state-of-the-art solutions designed for GAN training with limited data on StyleGAN2 [21]. We confine the data amount of FFHQ [20] and directly use AFHQ-Cat [9] that is small by itself. of-the-art approaches designed for the low-data regime, including ADA [19] and LC-regularization (LC-Reg) [46], which perform standard data augmentations and model regularization, respectively. In Figure 11, we show the qualitative results for further illustration. The synthesis quality of the StyleGAN2 [21] baseline deteriorates on the limited amount of training data. Ripple artifacts and substantial distortions appear on the generated images by StyleGAN2. LC-Reg [46] slightly improves the quality of synthesis by reducing the distortions on the images. The amelioration of visual quality is more evident by applying ADA [19], where the ripple artifacts are clearly subsided while some minor artifacts exist on the hair and beard. The proposed APA achieves comparable or even better visual quality than LC-Reg [46] and ADA [19], effectively improving the StyleGAN2 synthesized results on limited data. Notably, APA is also complementary to ADA [19] for gaining a further performance boost, suggesting the compatibility of our method with standard data augmentations. The visual results we present here are in line with the quantitative performance in our main paper.
More comparative results with ADA [19]. Table 9 reports additional comparative results with the state-of-the-art ADA [19] tailored for GAN training with limited data to further highlight the StyleGAN2 APA (Ours) Figure 12: The effectiveness of our method to improve state-of-the-art StyleGAN2 [21] higherresolution synthesized results (1024 × 1024, no truncation) trained with the limited data, i.e., FFHQ-5k [20] (a subset of 5, 000 images, ∼ 7% of full data). Our method achieves an FID score of 9.545, outperforming the original StyleGAN2 of 18.296. advantages of the proposed APA. We include additional comparisons on FFHQ-7k [20] and FFHQ-1k (see our main paper for 5k and 70k). To make the comparison settings more comprehensive, we also provide the transfer learning results on MetFaces [19] from the pre-trained StyleGAN2 model on FFHQ-70k. We also make a comparison with a fewer data amount (i.e., 500 images). Combined with the results we presented in the main paper, the proposed APA achieves comparable or even better performance than ADA [19] while with less computational cost. Both methods outperform the StyleGAN2 baseline under limited data. Although applying APA solely may be inferior to ADA [19] on FFHQ-1k (in line with our discussion in the main paper), it is worth mentioning that APA is also complementary to ADA [19], which is very important to boost the performance further.
Higher-resolution examples on StyleGAN2. In Figure 12, we show some higher-resolution (1024× 1024) synthesized images on FFHQ-5k [20] (a subset of 5, 000 images, ∼ 7% of full data) to further illustrate the effectiveness of our approach in improving StyleGAN2 with limited training data. The truncation trick [6,20,21] is not applied. The proposed APA evidently improves StyleGAN2 in both perceptual quality and the FID score, indicating its effectiveness for GAN training with limited data.
Evolution of the deception probability. In Figure 13, we visualize the evolution of the deception probability p in training on FFHQ [20] (256 × 256) with different data amounts. The evolution of p may be relatively more unstable when the data amount is very limited (i.e., 1k). Notably, the proposed APA possesses a desired property that the deception probability p can be naturally restricted under a safe limit (∼ 0.8) regardless of the training data amounts. Hence, the fundamental capability of discriminator in adversarial training may be better preserved thanks to the adaptive control scheme.
More overfitting and convergence analysis. Aside from the overfitting and convergence analysis on FFHQ-7k (10% of full data) in our main paper, we present the additional analysis on FFHQ-5k (∼ 7% of full data) and FFHQ-1k (∼ 1.4% of full data) in Figure 14 and Figure 15, respectively. The trend of model overfitting and convergence on FFHQ-5k and FFHQ-1k is similar to FFHQ-7k. The discriminator raw output logits of instance noise (ISN) [44] and APA. (c) The discriminator raw output logits of one-sided label smoothing (OLS) [42] and APA.  The divergence of StyleGAN2 discriminator predictions can be effectively restricted by applying the proposed APA, indicating its effectiveness in mitigating the discriminator overfitting. Meanwhile, APA improves the training convergence measured by FID.
Furthermore, we show the comparative overfitting analysis on APA and previous techniques for regularizing GANs in Figure 16. We observe that the effectiveness to counteract overfitting is in line with the generation performance of these methods in the main paper. Specifically, StyleGAN2 experiences diverged predictions most rapidly, and APA obtains the most effective restriction on the divergence of discriminator outputs. Applying instance noise (ISN) [44] produces curves that are very close to StyleGAN2, and one-sided label smoothing (OLS) [42] only restricts the real predictions. This further suggests: 1) the importance of addressing the discriminator overfitting for training GANs with limited data; 2) the effectiveness of APA in alleviating the discriminator overfitting, outperforming previous strategies.