{'fraction_non_alphanumeric': 0.03846645367412141, 'fraction_numerical': 0.012864749733759319, 'mean_word_length': 4.681510164569216, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Generative Adversarial Networks via a Composite Annealing of Noise and Diffusion

Generative adversarial network (GAN) is a framework for generating fake data using a set of real examples. However, GAN is unstable in the training stage. In order to stabilize GANs, the noise injection has been used to enlarge the overlap of the real and fake distributions at the cost of increasing variance. The diffusion (or smoothing) may reduce the intrinsic underlying dimensionality of data but it suppresses the capability of GANs to learn high-frequency information in the training procedure. Based on these observations, we propose a data representation for the GAN training, called noisy scale-space (NSS), that recursively applies the smoothing with a balanced noise to data in order to replace the high-frequency information by random data, leading to a coarse-to-fine training of GANs. We experiment with NSS using DCGAN and StyleGAN2 based on benchmark datasets in which the NSS-based GANs outperforms the state-of-the-arts in most cases.


Introduction
Generative adversarial network (GAN) [1] is a machine learning framework to generate realistic fake data. GAN learns the probabilistic distribution of the training (real) data using two adversarial networks: the generator that is trained to create realistic fake data from a random seed called the latent vector, and the discriminator that is dedicated to distinguish the reals against fake data.
GAN has been studied extensively in the several past years and currently is an essential tool for a wide variety of applications.
However, the training procedure is prone to numerical unstability in GANs [2,3,4,5,6]. Since it is a two-player game between the generator and discriminator [4], the optimization can fall into a local minima where the discriminator reaches a perfect solution first and the generator cannot be trained anymore.
This failure of GAN severely limits the quality of fake data, resulting in the mode collapse. The failure of GAN occurs when there is almost no overlap between the real distribution with fake distribution [7], in particular, in the beginning of training when the discriminator can reject fake data with a high confidence [1]. Therefore, GANs require stabilization methods in the optimization process in order to obtain better fake data.
The stabilization methods of GAN are different with those for the feedforward deep networks, e.g., weight-decay [8] and the momentum [9], due to the dynamics of the discriminator with generator. Since the failure of GAN can arise from the use of KL-divergence in loss, a variety of the discrepancy measures have been studied, e.g., [10,11,12], including Wasserstein distance [7,13] that provides gradients to the generator even with a small overlap of the two distributions. Albeit, the Wasserstein loss is often inferior to the original loss in the quality of fake data [3,5]. Another strategy is to inhibit discriminator training based on the loss. To this aim, the gradient regularization penalizes gradients of the discriminator. However, it depends on the model that varies during the training [5].
This paper focuses on data-based stabilization of the GAN training that manipulates only the real and fake data independent of the model architecture.
For instance, the repetition of noise injection to both real and fake data (namely noise-space) enlarges the overlap between probability distributions [14]. However, the noise increases the variance of data inevitably. The repetition of data smoothing based on Gaussian kernels, or the scale-space, is a general technique in machine learning that suppresses the high frequency features such as textures and details in the image so as to make each data more simple to learn by algorithms. In the case of GAN, it is expected that the diffusion makes the real data easy to mimic by the generator. However, we have found that it limits the learning capability of the generator to learn high-frequency information.
Based on these observations, we propose an algorithm for stabilizing the optimization of GANs based on a noisy scale-space (NSS) that continuously removes high-frequency information in image while adding noise. The proposed noisy scale-space enables a coarse-to-fine training of GANs in which we can train a generative model using low-level information in data with noise without the increment of data variance, while keeping the current model capable of learning the high-level information. We also present a synthetic dataset using the Hadamard bases [15] that can visualize the true distributions of real and fake data in order to characterize the drawback of the conventional scale-space in GAN optimization. Then, we perform experiments with the proposed NSS using DCGAN [16]. The experimental results based on the major datasets show that the proposed NSS-GAN outperforms other methods in most cases irrespective of the image generation tasks. Specifically, it is shown that the stabilization effect by our method is not the simple summation of the noise-space with the scalespace but due to the use of their mutually complementary relationship. We also demonstrate that the proposed NSS can improve the accuracy of StyleGAN2 [17] for high-resolution images.
We relate our method to prior works in Section 2. Then we consider a data-based stabilization of GAN training in Section 3, followed by the proposed noisy scale-space in Section 4. The effectiveness of our method is demonstrated experimentally in Section 5 and we conclude in Section 6.

Related works
Loss-based GAN stabilization: The regularization term in the loss has been studied to restrict the update of discriminator. In a related context, Wasserstein-GAN (WGAN) uses a weight clipping [13] to guarantee the Lipschitz constraint. WGAN-gp uses a gradient penalty [18] that can improve the quality of fakes in practice. The spectral normalization [19] is an efficient variant of the gradient-penalty. Dragan [20] penalizes the sharp gradient of discriminator to real data. We will use the non-saturating loss [1] based on the KL-divergence that is known to be the best choice in practice. The nonsaturating loss can be further improved using the gradient regularization [3] that penalizes the gradient norm of the discriminator. It is shown that the penalty based on the gradient-norm of the discriminator is equivalent to adding input noise in GAN using f -divergence [21]. The drawback of gradient regularization is that it depends on the distribution of fakes determined by the generator which changes during training [5].
Procedure-based GAN stabilization: The two time-scale update rule [22] is a method of using different annealings for discriminator and generator in order to slow the convergence of the discriminator. The simultaneous update of the two networks was studied in [23]. Progressive augmentation of GAN [24] extends the label noise [25] into the GAN framework to perturb the real and fake labels. The one-sided label smoothing replaces the 0 and 1 target labels for the discriminator with smoothed values, like 0.9 or 0.1 [18,26].
Data-based GAN stabilization: The proposed method can be categorized in data-based methods that manipulate only data. Lens-GAN [27] introduces a filtering network that transforms the real data against the discriminator, and therefore it still depends on the networks.
The multi-resolution training of GAN is a topic studied in, e.g., Progressive-GAN [28] and MSG-GAN [29], for generating high-resolution images. It trains a shallow network first using low-resolution images, and gradually increases both the number of network layers and the resolution of data. The drawback of the multi-resolution training is that it strongly limits the architecture.
The data augmentation is another recent topic in GAN training [30,17,31] in which a multitude of data transformations, typically consist of spatial transformations (e.g., image rotation, flipping, and cropping) with color transformations (e.g., channel permutation and hue rotation) are combined and applied to both real and fake data in order to enlarge the variation of data. The early works [30,17] are oriented to avoid over-fitting of GANs to a small set of training examples while the recent works aim to improve the accuracy as we do. Also DistAug [32] has considered a mixture of data transformations in a contrastive training of GANs [33]. However, the mixture of transformations in these studies is a black-box. In contrast, we present a deep understanding of the noise injection and the image diffusion in GAN optimization and then propose a better use of their mutually complementary relationship.
Our method is closely related to the noise injection and the data smoothing.
On one hand, the noise injection flattens the probability distribution of data.
Thus imposing noise to the real data [34,7] or both the real and fake data [14] enlarges the overlap of the two distributions. However, adding high-dimensional noise introduces significant variance in the parameter estimation, slowing down the training and requiring multiple samples for counteraction [21,24]. On the other hand, the scale-space decreases the dimensionality of data by removing high-frequency information and makes data easy to learn by algorithms [35,36]. However, the conventional scale-space has a critical side-effect in GAN optimization. Different from these baselines, we present a data representation that mitigates the side effects of the noise injection and the data smoothing.
Our algorithm also relates to Ambient-GAN [37] that considers a problem in which incomplete real data containing a Gaussian blurring with additive noise are given. However, Ambient-GAN aims to approximate the original distribution from the incomplete measurements using a conditional network, and created fake images are degenerated by the noise and smoothing. In contrast, we propose a continuous data representation to train GANs, achieving a better quality of fake data than the baseline method using the complete examples.

Generative adversarial networks
Let us begin with a technical introduction to the generative adversarial networks (GAN). Given a set of real data (x), GAN aims to generate new data with similar statistics as the real data. GAN consists of the two networks: the generator (G) that creates fake data (G(z)) from a random latent vector (z), and the discriminator (D) that distinguishes the real data against the fake data.
To this aim, a min-max loss (F ) is defined [1] as where p data (x) is the true distribution of the reals, and p z (z) is a random probability distribution. GANs are optimized using a stochastic gradient descent, e.g., Adam [38], in an alternative way such that the generator is updated to minimize F while the discriminator is trained to maximize F .

Data-based stabilization of GAN training
We consider a data-based stabilization method, similar to [30,17,31], that projects both the reals and fakes to a data space and feed into the GAN loss as where Φ t (x) and Φ t (G(z)) are the projected data of reals and fakes using a function Φ with the time parameter t. Using Eq.(2), the discriminator is trained to distinguish the projected real against the projected fake while the generator is trained to create fake data such that the projected fakes Φ t (G(z)) are similar to the projected reals Φ t (x). The data-based stabilization effect will be imposed via a discrete representation of data where Φ t (y) with larger t is expected to have larger effect such that the data is more easy to learn by GANs, and Φ 0 (y) = y denotes the original real or fake data. To this aim, the time parameter (t) should start with a large value T and shrink to zero during the training, and the choice of Φ determines the stabilization effect.

Proposed noisy scale-space
We present a data representation, namely noisy scale-space (NSS), that is designed to improve the stability of the optimization for GANs while preserving characteristic features in the generation process. The presented data representation is a balanced composition of the noise with the diffusion such that we can train GANs using the smoothed data that is easy to create by the generator first, while flattening the fake distribution in the high-frequency domain, making the current solution be capable of learning the high-frequency information in the further steps. Formally, the noisy scale-space is given as where k is typically the 3 × 3 Gaussian kernel, the symbol * denotes the convo-

Comparison to conventional noise-space
Given a constant σ, we can obtain the conventional noise-space as with t ∼ N (0, σ), t ∈ [T ], and Φ n 0 (y) = y. Equation (4) provides a discrete representation of data with additive noises. Figure 1 (a) shows an example of noise-space using a real image. We refer to GAN trained using the noise-space as Noise-Space (NS) GAN. Considering the sum of normal distributions, the noise-space defined by Eq.(4) is equivalent to . Therefore, the drawback of the noise-space is that it increases the variance of data, and makes the original data harder to be learned.
Note that the proposed noisy scale-space using Eq. (3) can be rewritten in a closed-form as where k * (t) denotes the t-times convolution with kernel k, and k * (0) t = t . Thus, our noisy scale-space balances the variances of the data-term and the noise-terms using the smoothing. A finding is that we can keep the data variance using a constant σ as demonstrated in Figure 2. Equation (6) also shows that the noisy scale-space replaces the high-frequency information in data by a set of noises with the corresponding frequencies. We demonstrate this property using a synthetic dataset in the following section. Table 1: Fréchet inception distance (FID) and inception score (IS) for Celeba dataset by DCGAN trained using (1st, ..., 7th columns) two-step scale-space {t, 0} with equal periods using t = 0, 2, ..., 128, respectively, and (8th column) a scale-space using exponential annealing with t = 128: t = 0 is the baseline. The mean of FID and IS were computed within 20 trials.

Comparison to conventional scale-space
The conventional scale-space is a multi-scale representation of image in which the smoothing is applied to data recursively as with Φ s 0 (y) = y and t ∈ [T ]. As shown in Figure 1 (b), the scale-space continuously removes the low-level information, e.g., textures and details, and provides the high-level information that is invariant to the scale.
We first re-examine the effect of diffusion in GAN optimization. Figure 3 visualizes the prediction curves of DCGAN [16] based on CelebA [39] where we trained the networks using the fixed smoothing time (t) applied to both real and fake data. We have chosen DCGAN since the convolution networks is the essential architecture of the modern GANs, e.g., [42,43,44,45,46,47,48]. As demonstrated in Figure 3, a larger t results in better prediction curves where both D(Φ s t (x)) and D(Φ s t (G(z))) converge to 0.5. Thus, the diffusion will make the problem more easy to learn by GANs. Note that the quality of G(z) degenerates with t since there is information loss due to the diffusion. Therefore, it is natural to anneal t over the training process.
However, the conventional scale-space cannot improve the accuracy of GANs in general. Table 1 summaries the Fréchet inception distance (FID) [22] and the inception score (IS) [18] by DCGAN for CelebA [39] using scale-spaces, where we first used a fixed smoothing t for 5 epochs then used the original data (t = 0) for 5 epochs. More details about the experimental set-up are summarized in Section 5.1. As shown in Table 1, GANs using scale-spaces are often inferior to the baseline without smoothing.
Our key observation is that the scale-space has a side effect in the GAN optimization where the smoothed data (real and fake data) make the generator create a smoothed data and thus shrink the capability of the generator to learn ). This means that the data smoothing reduces the overlap between the fake distribution with the distribution of reals with fine details that will be given in further steps.
We visualize the effect of the noisy scale-space using the synthetic dataset in Figure 5 in which we used a fixed t = 8. Figure 5 shows that (left) the projected data are smoothed yet (right) the diversity of high-frequency coefficients of fakes are preserved as expected.
Here, we have studied the side-effect of the diffusion. Interestingly, this kind of phenomenon due to a data transformation in GAN optimization is called 'leaking' that has been studied on, e.g., rotation and hue change [17]. The contributions of our work are that we visualize the leaking of the smoothing using the synthetic data; and we propose the prescription to mitigate the sideeffect of diffusion in GAN training.

Implementation of noisy scale-space
The data-based stabilization methods can be embedded into GANs as depicted in Figure 6 (a). We implement our data representation into DCGAN [16] that is the foundation of recent extensive studies, and call it Noisy Scale-Space where i ∈ [0, 1] is the relative iteration in the optimization process that starts at i = 0, β is the power of decay, and T is the initial time t (0) . We use β = 20 in order to apply the data transformation in the early stage of GAN optimization.
Figure 6 (c) shows the annealing curve of t with T = 256. We have observed that the exponential function achieves better accuracy than others including the step function and the decaying-wave function. Also we employ an implementation technique [14] in which we apply the filtering function Φ to only half of data in real-and fake-batches for obtaining a stable and accurate generator.

Experimental results
In order to empirically demonstrate the expected stabilization effect of the noisy scale-space for GAN optimization, we conduct three experiments: an ablation study on the initial scale in data spaces, a primal experiment in which we compare the proposed NSS with the state-of-the-art GANs based on DCGAN, and an additional experiment using StyleGAN2.

Experimental set-up for primal experiments
In the preliminary experiment, we compare our NSS-GAN with potential We use the non-saturating loss that is known to be superior to the Wasserstein loss and others [3], and employ Adam [38]  For quantitative evaluation of generated fake images, we use the Fréchet inception distance (FID) [22] with the inception score (IS) [18] that are widely used in GAN studies. FID measures the distance between the real data with fake data in feature space defined using a pre-trained network. IS measures the diversity of fakes in the feature space. Lower FID values with higher IS indicate better quality and diversity of fake data, respectively. We use FID as the primal metric that reflects the objective of GAN.    Table 2 shows the mean FID within the 20 trials over the initial time of T = 0, 32, 64, 128, 256 where T = 0 is the baseline DCGAN. Table 2 [49], WGAN-gp [18], (bottom) Dragan [20], and GANs using the scale-space (SS), the noise-space (NS), and the proposed noisy scale-space (NSS) with T = 256.   [3], LSGAN [49], WGAN-gp [18], Dragan [20], and GANs using the

Comparison to state-of-the-arts
We now compare the proposed NSS-GAN with the state-of-the-arts of GANoptimization: the baseline DCGAN (GAN), GAN with the gradient regularization [3], LSGAN [49], WGAN with gradient penalty (WGAN-gp) [18], Dragan [20], GANs using the scale-space (SS-GAN), and the noise-space (NS-GAN) using the four of datasets. Note that these GANs share the same backbone architecture of DCGAN. In order to make our result independent to the hyper parameters of Adam, we employ grid search and tuned the learning-rate scale (η) in combination with the first momentum coefficient (b 1 ) for each pair of model and dataset, while the second momentum of Adam was fixed as b 2 = 0.999 based on our pretest. For each model, we then choose the best condition using the mean FID within the 20 trials. Table 3 summarizes the tuned parameters in which the baseline GAN prefers a stable condition compared to the others. More importantly, Table 4 shows that the conventional scale-space (SS) GAN and the noise-space (NS) GAN were not consistently better than the baseline GAN. In contrast, the proposed NSS-GAN has consistently outperformed the baseline GAN, SS-GAN, and NS-GAN in FID. This indicates that the stabilization effect of the presented NSS-GAN is not the simple summation of those by the scale-space with the noise-space but due to the better use of their mutually complementary relationship in the GAN optimization.

Comparison to StyleGAN2-Ada
As an additional study, we apply the noise scale-space to StyleGAN2-Ada [17] and compare it with the original algorithm using MetFaces dataset [17] and We follow the experimental set-up, the evaluation metrics, and the hyperparameters of the official StyleGAN2 implementation, including the R1 regularization weight (γ) that we have tuned within γ = 0.82, 1.64, 3.28 as recommended in [17]. We tune the noise std of NSS based on the image variance curve of the images and also employ T = 128 as the initial condition. Table 5 summarizes the tuned hyper-parameters. Table 6 summarizes FID and IS by StyleGAN2-Ada and StyleGAN2 using the proposed noisy scale-space. Figure 9 visualizes fake images created by the generator with the average FID within the individual trials. Table 6 and Figure 9 show that the presented NSS-GAN has successfully improved the accuracy and quality of generated images using the state-of-art StyleGAN2.
It has been reported [17] that StyleGAN2-Ada outperforms Progressive-GAN [28] in accuracy. Therefore, our experimental results implicitly demonstrate that StyleGAN2-NSS is superior to Progressive-GAN. Moreover, our noisy scalespace is independent of the architecture in contrast to Progressive-GAN that requires the hierarchical architecture of networks. StyleGAN2 with the proposed noisy scale-space. The models were trained 5KImg-iterations.
The mean and std of the metrics were computed within the individual 5 trials.

Conclusion
In the consideration of data manipulation for the stable optimization in GANs, we have proposed a discrete representation of data, called noisy scalespace (NSS), that gradually removes high-frequency information in image while adding noise, leading to a coarse-to-fine training of GANs. In order to observe the side-effect of the conventional scale-space in GAN optimization, we have proposed the synthetic dataset based on the Hadamard bases that visualizes the true distribution of the real and fake data. We have experimented with the proposed NSS using two backbone networks: DCGAN and StyleGAN2 based on the major datasets for natural image generation tasks. The experimental results have successfully demonstrated that: NSS-based GANs overtook the potential competitors and the state-of-the-arts in most cases.

StyleGAN2-Ada
StyleGAN2-NSS (ours) A limitation of our method is that we assume the diffusion can simplify (the real) data. Concretely, our NSS-GAN is inferior to the original GAN when using MNIST [50] images that consist of 0/1 binary values. Obviously, smoothing the binary data increases the diversity of pixel values. Our assumption holds for natural images and our method yields the sufficient stabilization effect for GAN optimization irrespective of the content of images.