{'fraction_non_alphanumeric': 0.04233664424747227, 'fraction_numerical': 0.01129125651418645, 'mean_word_length': 4.724502804691484, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improved Input Reprogramming for GAN Conditioning

We study the GAN conditioning problem, whose goal is to convert a pretrained unconditional GAN into a conditional GAN using labeled data. We first identify and analyze three approaches to this problem -- conditional GAN training from scratch, fine-tuning, and input reprogramming. Our analysis reveals that when the amount of labeled data is small, input reprogramming performs the best. Motivated by real-world scenarios with scarce labeled data, we focus on the input reprogramming approach and carefully analyze the existing algorithm. After identifying a few critical issues of the previous input reprogramming approach, we propose a new algorithm called InRep+. Our algorithm InRep+ addresses the existing issues with the novel uses of invertible neural networks and Positive-Unlabeled (PU) learning. Via extensive experiments, we show that InRep+ outperforms all existing methods, particularly when label information is scarce, noisy, and/or imbalanced. For instance, for the task of conditioning a CIFAR10 GAN with 1% labeled data, InRep+ achieves an average Intra-FID of 76.24, whereas the second-best method achieves 114.51.

Conditioning has become an essential research topic of GANs. While earlier works focus on unconditional GANs (UGANs), which sample data from unconditional data distributions, conditional GANs (CGANs) have recently gained a more significant deal of attention thanks to their ability to generate high-quality samples from class-conditional data distributions [2,17,18]. CGANs provide a broader range of applications in conditional image generation [17], text-to-image generation [19], image-to-image translation [5], and text-to-speech synthesis [20,21].
In this work, we define a new problem, which we dub GAN conditioning, whose goal is to learn a CGAN given (a) a pretrained UGAN and (b) labeled data. The pretrained UGAN is given in the form of an unconditional generator. Also, we assume that the classes of the labeled data are exclusive to each other, that is, the true class-conditional distributions are separable. Fig. 1 illustrates the problem setting with a two-class MNIST dataset. The first input, shown on the top left of the figure, is an unconditional generator G, which is trained on a mixed dataset of classes 0 and 1. The second input, shown on the bottom left of the figure, is a labeled dataset. In this example, the goal of GAN conditioning algorithms is to learn a two-class conditional generator G from these two inputs, as shown on the right of Fig. 1. Each Dy is built on a weight-sharing network D followed by a class-specific linear head Hy, i.e. Dy = Hy • D. We embed each label y into a vector y with the embedding module E, then concatenate y to a random noise vector u to get vector z. The modifier M converts z into a y-conditional noise zy so that G(zy) is a sample of class y (a 0-image of fakey in the illustration). We train our networks using GAN training.
INREP+. In Sec. 4, we empirically evaluate INREP+ and other GAN conditioning methods in various training settings. Finally, Sec. 5 provides further discussions on the limitation of our proposed approach as well as the applicability of INREP+ to other generative models and prompt tuning. We conclude our paper in Sec. 6.
Notation We follow the notations and symbols in the standard GAN literature [1,34]. That is, a, a, a, a, A denote a scalar, vector, random scalar variable, random vector variable, and set; x, y, z denote a feature vector, label, and input noise vector, respectively.

Preliminaries on GAN conditioning
We review three existing approaches to GAN conditioning: (1) discarding UGAN and training a CGAN from scratch, (2) fine-tuning [25,26], and (3) input reprogramming [28,29]. In particular, we analyze their advantages, drawbacks, and requirements for GAN conditioning. Sec. 2.4 summarizes our high-level comparison of their performances in various training settings of labeled data and computing resources.

Learning CGAN without using UGAN
One can discard the given UGAN and apply the existing CGAN training algorithms on the labeled dataset to learn a CGAN. This approach is the most straightforward approach for GAN conditioning.
We can group CGAN training algorithms by strategies of incorporating label information into the training procedure. The first strategy is to concatenate or embed labels to inputs [17,35] or to middle-layer features [19,36]. The second strategy, which generally achieves better performance, is to design the objective function to incorporate conditional information. For instance, ACGAN [18] adds a classification loss term to the original discriminator objective via an auxiliary classifier. Recent algorithms adopt label projection in the discriminator [2] by linearly projecting the embedding of the label vector into the feature vector. Based on the projection-based strategy, ContraGAN [4] further utilizes the data-data relation between samples to achieve better quality and diversity in data generation.
Directly applying such CGAN algorithms for GAN conditioning may achieve state-of-the-art conditioning performances if sufficient labeled data and computing resources are available. However, this is not always the case in practice, and the approach also has some other drawbacks. The scarcity of labeled data in practice can severely degrade the performance of CGAN algorithms [24]. Though multiple techniques [3,37] were proposed to overcome this scarcity issue, mainly by using data augmentation to increase labeled data, these techniques are orthogonal to our work, and we consider only the standard algorithms. Furthermore, training CGAN from scratch is notoriously challenging and expensive. Researchers observed various factors that make CGAN training difficult, such as instability [22,23], mode collapse [38,23], or mode inventing [38]. Also, the training usually entails enormous resources in terms of time, computation, and memory, which are not available for some settings, such as mobile or edge computing. For instance, training BigGAN [39] takes approximately 15 days on a node with 8x NVIDIA Tesla V100/32GB GPUs [16].
Additionally, some CGAN training algorithms have inductive biases, which may cause failures in learning the true distributions. The following lemmas investigate several failure scenarios of the two most popular conditioning strategies -ACGAN [18] and ProjGAN [2], and we defer the formal statements and proofs to Appendix A.1.
Failures of auxiliary classifier conditioning strategy The discriminator and generator of ACGAN learn to maximize λL C + L S and λL C − L S , respectively. Here, L S models the log-likelihood of samples belonging to the real data, L C models the log-likelihood of samples belonging to the correct classes, and λ is a hyperparameter balancing the two terms. In this game, G might be able to maximize λL C − L S by simply learning a biased distribution (increased L C ) at the cost of compromised generation quality (increased L S ). Our following lemmas show that ACGAN indeed suffers from this phenomenon, for both non-separable datasets (Lemma 1) and separable datasets (Lemma 2). We also note that the failure in non-separable datasets has been previously studied in [40], while the one with separable datasets has not been shown before in the literature. Lemma 1 (ACGAN provably fails on a non-separable dataset). Suppose that the data follows a Gaussian mixture distribution with a known location but unknown variance, and the generator is a Gaussian mixture model parameterized by its variance. Assume the perfect discriminator. For some values of λ, the generator's loss function has strictly suboptimal local minima, thus gradient descent-based training algorithms fail to find the global optimum. Lemma 2 (ACGAN provably fails on a separable dataset). Suppose that the data are vertically uniform in 2D space: conditioned on y ∈ {±1}, x = (d · y, u) with some d > 0 and u is uniformly distributed in [−1, 1]. An auxiliary classifier is a linear classifier, passing the origin. Assume the perfect discriminator. For some single-parameter generators, gradient descent-based training algorithms converge to strictly suboptimal local minima.
Failures of projection-based conditioning strategy The objective function of a projection-based discriminator [2] measures the orthogonality between the data feature vector and its class-embedding vector in the form of an inner product. Thus, even when the generator learns the correct conditional distributions, it may continue evolving to further orthogonalize the inner product term if the class embedding matrix is not well-chosen. This behavior of the generator can result in learning an inexact conditional distribution, illustrated in the following lemma. Lemma 3 (ProjGAN provably fails on a two-class dataset). Consider a simple projection-based CGAN with two equiprobable classes. With some particular parameterizations of the discriminator, there exist bad class-embedding vectors that encourage the generator to deviate from the exact conditional distributions.

Fine-tuning UGAN into CGAN
The fine-tuning approach aims to adjust the provided unconditional generator into a conditional generator using the labeled data. Previous works [25,26] studied fine-tuning GANs for knowledge transfer in GANs. We can further adapt these approaches for GAN conditioning. TransferGAN [25] introduces a new framework for transferring GANs between different datasets. They propose to fine-tune both the pretrained generator and discriminator networks. MineGAN [26] later suggests fixing the generator and training an extra network (called the miner) to optimize the latent noises for the target dataset before fine-tuning all networks. We note that in the GAN conditioning setting, the pretrained discriminator is not available. Furthermore, to be applicable for GAN conditioning, TransferGAN and MineGAN frameworks may require further modifications of unconditional generators' architecture to incorporate the conditional information. Unlike fine-tuning approaches, our method freezes the G and does not modify its architecture or use the pretrained unconditional discriminator.
Compared to the full CGAN training, fine-tuning approaches usually require much less training time and amount of labeled data while still achieving competitive performance. However, it is not clear how one will modify the architecture of the unconditional generator for a conditional one. Also, fine-tuning techniques are known to suffer from catastrophic forgetting [27], which may severely affect the quality of generated samples from well-trained unconditional generators.

Input reprogramming
The overarching idea of input reprogramming is to keep the well-trained UGAN intact and add a controller module to the input end for controlling the generation. In particular, this approach aims to repurpose the unconditional generator into a conditional generator by only learning the class-conditional latent vectors.
Input reprogramming can be considered as neural reprogramming [41] applied to generative models. Neural reprogramming (or adversarial reprogramming) [41] has been proposed to repurpose pretrained classifiers for alternative classification tasks by just preprocessing their inputs and outputs. Recent works extend neural reprogramming to different settings and applications, such as the discrete input space for text classification tasks [42] and the black-box setting for transfer learning [43]. More interestingly, it has been shown that large-scale pretrained transformer models can also be efficiently repurposed for various downstream tasks, with most of the pretrained parameters being frozen [44]. On the theory side, the recent work [45] shows that the risk of a reprogrammed classifier can be upper bounded by the sum of the source model's population risk and the alignment loss between the source and the target tasks.
Input reprogramming has been previously studied under various generative models and learning settings. Early works attempt to reprogram the latent space of GAN with pretrained classifiers [46] or the latent space of variational autoencoder (VAE) [28] via CGAN training. Flow-based models can also be repurposed for the target attributes via posterior matching [47]. The recent work [48] proposes to formulate the conditional distribution as an energy-based Table 1: A high-level comparison of GAN conditioning approaches under various settings ( = poor, = okay, = good). When labeled data and computing resources are both sufficient, we can discard the unconditional generator and directly apply CGAN algorithms on the labeled data to achieve the best performances. When the amount of labeled data is large, but resources are restricted, the fine-tuning approach becomes a good candidate for GAN conditioning. However, when both labeled data and resources are restricted, input reprogramming becomes the best solution for GAN conditioning. Our method INREP+ improves both the conditioning performance and the memory scalability of the existing input reprogramming algorithm (GAN-REPROGRAM).

Setting
Learning CGAN w/o UGAN Fine-tuning Input reprogramming GAN-REPROGRAM INREP+ (ours) Large labeled data Limited labeled data Limited computation model and train a classifier on the latent space to control conditional samples of unconditional generators. However, their training requires manually labeling latent samples and a new sampling method based on ordinary differential equation solvers. We also notice that our work has a different assumption on the input as the only source of supervision in GAN conditioning comes from the labeled data. Recently, GAN-REPROGRAM [29] via CGAN training obtains high-quality conditional samples with significant reductions of computing and label complexity. GAN-REPROGRAM is our closest input reprogramming algorithm for GAN conditioning.
For the advantages, input reprogramming methods obtain the competitive performances in the conditional generation on various image datasets, even with limited labeled data. Input reprogramming significantly saves computing and memory resources as the approach requires only an extra lightweight network for each inference. However, the existing algorithm of input reprogramming for GAN conditioning, GAN-REPROGRAM [29], still underperforms the latest CGAN algorithms on complex datasets. In addition, it focuses solely on one condition per time, leading to the huge memory given a large number of classes, which will be detailed in the next section.
Our proposed method (INREP+) makes significant improvements on GAN-REPROGRAM. We improve the conditioning performance with invertible networks and a new loss based on the Positive Unlabeled learning approach. We reduce the memory footprints by sharing weights between class-conditional networks. Our study will exhibit that input reprogramming can generate sharper distribution even with small amounts of supervision, leading to consistent performance improvements. We also explore the robustness of input reprogramming in the setting of imbalanced supervision and noisy supervision, which have not been discussed in the literature yet. Table 1 summarizes our high-level comparison of the analyzed approaches. For input reprogramming, we distinguish between the existing approach GAN-REPROGRAM and our proposed INREP+. First, when both the labeled data and computation resources are sufficient, using CGAN algorithms without UGAN probably achieves the best conditioning performance, followed by the fine-tuning and INREP+ methods. However, when the labeled data is scarce (e.g., less than 10% of the unlabeled data), directly training CGAN from scratch may suffer from the degraded performance. Methods that reuse UGAN (fine-tuning, GAN-REPROGRAM, INREP+) are better alternatives in this scenario, and our experimental results show that INREP+ achieves the best performance among these methods. Furthermore, if the computation resources are more restricted, input reprogramming approaches are the best candidates as they gain significant computing savings while achieving good performances.

Improved Input Reprogramming for GAN conditioning
We first justify the use of the general input reprogramming method for GAN conditioning and analyze issues in the design of the current input reprogramming framework (Sec. 3.1). To address these issues, we propose a novel INREP+ framework in Sec. 3.2. We theoretically show the optimality of INREP+ in learning conditional distributions in Sec. 3.3. Algorithm 1 presents the full INREP+ training algorithm.

Revisiting input reprogramming for GAN conditioning
The idea of input reprogramming is to repurpose the pretrained unconditional generator G into a conditional generator simply by preprocessing its input without making any change to G. Intuitively, freezing the unconditional generator does not necessarily limit the input reprogramming's capacity of conditioning GANs. Consider a simple setting with the perfect generator G : G(z) = d x, where z and x are discrete random variables. For any discrete random variable y, possibly dependent on x, we can construct a random variable z y such that G(z y ) = d x|y = y by redistributing the probability mass of z into the values corresponding to the target x|y = y. We provide Proposition 2 in Appendix A.2 as a formal statement to illustrate our intuition in the general setting with continuous input spaces.
The input reprogramming approach aims at learning conditional noise vectors {z y } such that G(z y ) = d x|y for all y. We can view this problem as an implicit generative modeling problem. For each y, we learn a function M y (·) = M (·, y) such that z y = d M y (u) and G(z y ) = d x|y for standard random noise u via GAN training with a discriminator D y .
The existing algorithm for input reprogramming, GAN-REPROGRAM [29], proposes to separately learn M y , D y for each value of y, where M y is a neural network, called modifier network. Fig. 3 (left) visualizes the design of the GAN-REPROGRAM framework for an n-class condition. GAN-REPROGRAM requires n pairs of modifier and discriminator networks, each per class condition. This design has several issues in learning to condition GANs.
Issues of GAN-REPROGRAM in GAN conditioning First, there might be overlaps between the latent regions learned by modifier M y and M y (y = y) due to the imperfection in learning. These overlaps might result in incorrect class-conditional sampling. The second issue is the training dynamic of conditional discriminators. Assume the perfect pretrained unconditional generator, for instance, G(z) = d x with random Gaussian z and two equiprobable classes for simplicity. Then, we can view G(z) as a mixture of x|y = 0 and x|y = 1. At the beginning of (M y , D y ) training, a fraction of the generated data is distributed as x|y = y, which we desired, but labeled as fake and rejected by D y . As M y (u) approaches z y , a larger fraction of desired samples will be wrongly labeled as fake. This phenomenon can cause difficulty in reprogramming, especially under the regime of low supervision. The last issue is the vast memory when the number of classes is large because GAN-REPROGRAM trains separately each pair of (M y , D y ) per condition.

INREP+: An improved input reprogramming algorithm
We propose INREP+ to address identified issues of GAN-REPROGRAM. Shown in Fig. 2 is the design of our INREP+ framework. We adopt the weight-sharing design for modifier and discriminator networks to improve memory scalability. Also, we design our modifier network to be invertible to prevent the overlapping issue in the latent space. Lastly, we derive a new loss for more stable training based on the recent Positive-Unlabeled (PU) learning framework [33]. We highlight differences of INREP+ over previous GAN-REPROGRAM in Fig. 3.
Weight-sharing architectures For the modifier, we use only a single conditional modifier network M for all classes. For the discriminator, we share a base network D for all classes and design multiple linear heads H y on top of D, each head for a conditional class, that is, D y = H y • D. This design helps maintain the separation principle of input reprogramming while significantly reducing the number of trainable parameters and weights needed to store.
Invertible modifier network The modifier network M learns a mapping R dz → R dz y , where z is the input noise and z y is the y-conditional noise. The input noise z is an aggregation of a random noise vector u and the label y in the form of label embedding [2]. We use a learnable embedding module E to convert each discrete label into a continuous vector with dimension d y . The use of label embedding instead of one-hot encoding has been shown to help GANs learn better [2]. Here, d z = d u + d y with d z , d u , and d y being dimensions of z, u, and y respectively. In this work, we use the concatenation function as the aggregation function. To prevent outputs of modifiers on different classes from overlapping, we design the modifier to be invertible using the architecture of invertible neural networks [30,31,32]. Intuitively, the invertible modifier maps different random noises to different latent vectors, guaranteeing uniqueness. Notably, the use of invertibility leads to a dimension gap between the random noise u and the conditional noise z y as d u < d z = d zy . However, the small dimension gap does not significantly affect the performance of the modifier network on capturing the true distribution of the conditional latent because the underlying manifold of complex data usually has a much lower dimension in most practical cases [49]. Modifier M needs to be sufficiently expressive to learn the target noise partitions while maintaining the computation efficiency.
Training conditional discriminator with Positive-Unlabeled learning Assume that G generates y-class data with probability p class (y). At the beginning of discriminator training, p class (y) fraction of generated samples are high-quality and in-class but labeled as fake by the discriminator. This results in the training instability of the existing GAN reprogramming approach [29]. To deal with this issue, we view the discriminator training through Positive-Unlabeled (PU) learning lens. PU learning has been studied in binary classification, where only a subset of the dataset is labeled with one particular class, and the rest is unlabeled [50,51,52]. Considering the generated data as unlabeled, we can cast the conditional discriminator learning as a PU learning problem: classifying positive (in-class and high-quality) data from unlabeled (generated) data. Now, we consider training (M y , D y ) for a fixed y. As discussed, the distribution of generated data G(M y (u)) can be viewed as a mixture of p data (x|y) and the residual distribution, say p gf (x), where 'g' stands for generated and 'f' means that they should be labeled as fake because they are out-class. That is, letting π y be the fraction of y-class data among generated data, we have p G(My(u)) (x) = π y · p data (x|y) + (1 − π y ) · p gf (x). Given this decomposition, the ideal discriminator objective function is: (1) The discriminator loss is −V P U y . For real data x ∼ p data (x|y), discriminator minimizes a new loss −(1 + π y ) log(D y (x)) + π y log(1 − D y (x)) instead of the standard negative log-loss − log(D y (x)). Fig. 4 visualizes this new loss with various values of π y . Compared to the standard loss (π y = 0), our loss function strongly encourages the discriminator to assign positively labeled data higher scores that are very close to 1. Also, when the prediction is close to 1, the loss takes an unboundedly large negative value, indirectly assigning more weights to the first term in (2). If π y is close to 1, the relative weights given to the real samples become even higher. An intuitive justification is that the loss computed on the generated data becomes less reliable when π y gets larger, so one should focus more on the loss computed on the real data.
Note that the last term in (1) is always negative as D y outputs a probability value, but its empirical estimate, shown in the sum of the last two terms in (2), may turn out to be positive due to finite samples. We correct this by clipping the estimated value to be negative as shown below: We set π y = 1 |Y| initially when the generated data is well-balanced between classes, and gradually increase π to 1 by temperature scaling to capture the increasing fraction of y-class data. Remark 1. In the context of the UGAN training, recent work [33] studies a similar PU-based approach to address the training instability caused by the false rejection issue discussed above. However, unlike our approach, their approach does not consider the term E x∼pdata(x|y) [log D y (x)] in their loss function design. This design makes their loss function invalid when π y is close to 0, that is, at the beginning of GAN training. On the other hand, our loss function is valid for all π y ∈ [0, 1]. We also note that the false rejection issue is more relevant near the end of the standard GAN training when the generated samples are more similar to the real ones, while it occurs right at the beginning of GAN conditioning training. This difference leads to different uses of the PU-based loss for the standard GAN and GAN conditioning.
Algorithm 1: InRep+ Input: Pretrained unconditional generator G, class-conditional dataset D with label set Y, noise dimension d, batch size m, learning rates α, β, number of discriminator steps k, number of training iterations t Result: Class-conditional generator G Initialize parameters: θ M for modifier network M , θ E for embedding module E, θ D for discriminator network D, and {θ y : y ∈ Y} for linear heads {H y : y ∈ Y}.

INREP+'s optimality under ideal setting
In this section, we highlight that the GAN's global equilibrium theorem [1] still holds under the PU-learning principle incorporated in the discriminator training. Specifically, INREP+ attains the global equilibrium if and only if generated samples follow the true conditional distribution. This theorem guarantees that INREP+ learns the exact conditional distribution under the ideal training. We provide the proofs for the proposition and the theorem in Appendix A.3. Proposition 1. Fix the ideal unconditional generator G and arbitrary modifier M y , the optimal discriminator for y is D * y (M y (u)) = (1 + π y )p data (x|y) (1 + π y )p data (x|y) + (1 − π y )p gf (x) .
The proposition shows that the optimal discriminator D * learns a certain balance between p data (x|y) and p gf (x). Fixing such an optimal discriminator, we can prove the equilibrium theorem. Theorem 1 (Adapted from [1]). When the ideal unconditional generator G * and discriminator D * are fixed, the globally optimal modifier is attained if and only if p gf (x) = p G(My(u)) (x) = p data (x|y).
Evaluation metrics We use the popular Fréchet Inception Distance (FID) [57] and recall [58] to measure the quality and diversity of learned distributions. FID measures the Wasserstein-2 (Fréchet) distance between the learned and true distributions in the feature space of the pretrained Inception-v3 model. Lower FID indicates better performance. To measure the conditioning performance, we use Intra-FID [2] and Classification Accuracy Score (CAS) [59,60,61]. Specifically, Intra-FID is the average of FID scores measured separately on each class, and CAS is the testing accuracy on the real data of the classifier trained on the generated data.
Network architectures and training Our architectures and configurations of networks are mainly based on GANs' best practices [3,4,62,63]. We adopt the same network architectures for all models. For INREP+, we set the dimension of label embedding vectors to 10 for all experiments. Our modifier network uses the i-ResNet architecture [31], with three layers for simple datasets (Gaussian mixture, MNIST, CIFAR10) and five layers for more complex datasets (CIFAR100, FFHQ). We design the modifier networks to be more lightweight than generators and discriminators. For instance, our modifier for CIFAR10 has only 0.1M parameters while the generator and the discriminator have 4.3M and 1M parameters, respectively. The latent distribution is the standard normal distribution with 128 dimensions. For training, we employ Adam optimizer with the learning rates of 2 · 10 −4 and 2 · 10 −5 for modifier and discriminator networks. β 1 , β 2 are 0.5, 0.999, respectively. We train 100 000 steps with five discriminator steps before each generator step. INREP+ takes much less time compared to other approaches. For instance, training ACGAN and ContraGAN on CIFAR10 may take up to several hours, while training INREP+ takes approximately half of an hour to achieve the best performance. For implementation, we adopt the widely used third-party GAN library [4] with PyTorch implementations for reliable and fair assessments. We run our experiments on two RTX8000 (48 GB memory), two TitanX (11GB), two 2080-Ti (11GB) GPUs. For pretrained unconditional generators, we mostly pretrain unconditional generators on simple datasets (Gaussian mixture, MNIST, CIFAR10, CIFAR100), and make use of large-scale pretrained models for StyleGAN trained on FFHQ. 1 We provide more details in Appendix B.  (Fig. 5b), each row represents a class of generated conditional samples. We see that all ten rows have correct images with high quality and highly diverse shapes. Similarly, Fig. 5c shows two classes (wearing glasses and male) of high-resolution face images, which are generated using INREP+'s conditional generator. In this setting, UGAN is the StyleGAN model [55] pretrained on the unlabeled FFHQ data, and labeled data is the CelebA data. For CIFAR10, the last row of Fig. 6

GAN conditioning with different levels of supervision
We compare GAN conditioning methods under different amounts of labeled data. Specifically, we use different proportions of the training dataset (1%, 10%, 20%, 50%, and 100%) as the labeled data. Table 2 shows the comparison in terms of FID and Intra-FID on CIFAR10 and CIFAR100 datasets. Table 3 provides additional measures (recall and CAS scores) of the methods on CIFAR10. We also compare the quality of conditional samples on CIFAR10 in Fig. 6. We attribute good performances of INREP+ to its design that utilizes the representation of trained generators and efficiently uses limited labels in separating conditional latent vectors (Sec. 3.2). Interestingly, thanks to well-trained generators, GAN-REPROGRAM and fine-tuning also achieve better performances than other CGANs. For the qualitative comparison, Fig. 6 partly illustrates that most samples from INREP+ have higher image quality than those from CGAN methods and are more correctly conditioned than samples from fine-tuning and GAN-REPROGRAM.
As more labeled data is provided (20% and more), CGANs gradually obtain better scores and outperform INREP+, but INREP+ manages to keep relatively small gaps to the best CGANs. For instance, with full supervision on CIFAR10, these gaps between INREP+ and the best model, ProjGAN, are just 1.1 and 7.5 in terms of FID and Intra-FID.

Robustness to class-imbalanced data and noisy supervision
We analyze two practical settings of the labeled data, when the data is class-imbalanced and when the labels are noisy.

INREP+ outperforms GAN conditioning baselines on class-imbalanced CIFAR10
The class-imbalanced CI-FAR10 has two minor classes (0 and 1) that use 10% of their labeled data, while other classes remain unchanged. Training on this augmented dataset, we observe that INREP+ obtains the lowest overall FID (13.26), shown in Table 4. INREP+ also achieves the best Intra-FID scores for both two minor classes (69.48 and 61.68), which are close to the average Intra-FID of major classes (60.06). On the other hand, other methods show larger gaps between Intra-FID of minor classes and the average Intra-FID of major classes. For instance, these score gaps in ProjGAN and ACGAN are all greater than 20. This result validates the advantage of INREP+ in learning with class-imbalanced labeled data.
Remark 2 (Fairness in the data generation). The under-representation of minority classes in training data may bias models towards generating samples with more features of majority classes or samples of minority classes with lower quality and less diversity [64,65,66]. Being more robust against the class imbalance, INREP+ is a promising candidate for GAN conditioning with fair generation.

INREP+ is more robust to label noises in CIFAR10
We adopt the class-dependent noise setting in robust image classification [67,68], and corrupt clean labels with flipping probability 0.4 as follows: bird → airplane, cat → dog, deer → horse, truck → automobile. Shown in Table 5 are evaluated FID and Intra-FID. Though label corruption causes FID increases in all models, the increase of INREP+ is only 0.16 (compared to clean-label FID in Table 2), comparable to GAN-REPROGRAM and stark contrast with large gaps of CGAN methods. In terms of Intra-FID, INREP+ achieves the smallest score (63.15), and enjoys the smallest Intra-FID increases together with GAN-REPROGRAM (3.5 and 3.2) among evaluated methods. Also, their clean classes are less affected by label noises, while we observe non-negligible deviations in other models. These results show the strong robustness of input reprogramming methods against label noises. We attribute this robustness to their modular design, which preserves the representation learned by unconditional generators and focuses on learning the conditional distribution. Also, the multi-head design of the conditional discriminator contributes to preserving the separation between data of different classes. Thus, corrupted Table 4: FID evaluation on class-imbalanced CIFAR10. The dataset has 10 classes where classes 0 and 1 are minor. Compared to the FID on class-balanced data (Table 2), INREP+ and GAN-REPROGRAM obtain the smallest FID increases among methods, indicating that they are less affected by the imbalance in labels. Also, INREP+ achieves the best Intra-FIDs for minor classes, close to the average score of major classes, while the Intra-FID gaps between the minor and major classes are larger in other models.  Table 5: FID evaluation on label-noisy CIFAR10. The label-noisy CIFAR10 is constructed by corrupting labels of CIFAR10 with asymmetric noises. Compared to reported scores on the label-clean data (   We investigate the empirical effect of the invertible architecture and the PU-based loss on INREP+. We construct four INREP+ versions from enabling or disabling each of two components. We use ResNet architecture [69] and the standard GAN loss as replacements for the i-ResNet architecture and our PU-based loss. UGAN is pretrained on unlabeled CI-FAR10, and the labeled data is 10% of CIFAR10 (to observe better the effect of PU-based loss under the low-label regime). Shown in Table 6 are the Intra-FID scores of four models.

Discussion
Limitation INREP+ depends greatly on the pretrained unconditional generator: how perfectly the unconditional generator learns the underlying data distribution and how well its latent space organizes. The latest research [3,70] observes the existing capacity gaps between unconditional GANs and conditional GANs on large-scale datasets.
Therefore, how to improve INREP+ on large-scale datasets is still a challenging problem, especially when GANs' latent space structure has still not been well-understood. We leave this open problem for future work.
Beyond GAN: Can we apply INREP+ to other generative models? Our work currently focuses on GANs for studying GAN conditioning because GANs attain state-of-the-art performances in various fields [3,4,5,6,7,8], and numerous well-trained GAN models are publicly available. Nevertheless, INREP+ can be applied to other unconditional generative models. We can replace the unconditional generator from UGAN with one from other generative models, such as Normalizing Flows (Glow [71]) or Variational Autoencoders (VAEs) [72]. Interestingly, we notice that GAN training (as of INREP+) using generators pretrained on VAEs probably helps stabilize the training, reducing the mode collapse issue [73].
Enhancing input reprogramming to prompt tuning Text prompts, which are textual descriptions of downstream tasks and target examples, have been shown to be effective at conditioning the GPT-3 model [12]. Multiple works have been proposed to design and adapt prompts for different linguistic tasks, such as prompt tuning [74] and prefix tuning [75]. The latter approach freezes the generative model and learns the prefix activations in the encoder stack. Sharing similar merit to prompt tuning, INREP+ can be a promising solution for learning better prompts in conditioning generative tasks while achieving significant computing savings by freezing the generative model.

Conclusion
In this study, we define the GAN conditioning problem and thoroughly review three existing approaches to this problem. We focus on input reprogramming, the best performing approach under the regime of scarce labeled data. We propose INREP+ as a new algorithm for improving the existing input reprogramming algorithm over critical issues identified in our analysis. In INREP+, we adopt the invertible architecture for the modifier network and a multi-head design for the discriminator. We derive a new GAN loss based on the Positive-Unlabeled learning to stabilize the training. INREP+ exhibits remarkable advantages over the existing baselines: The ideal INREP+ training provably learns true conditional distributions with perfect unconditional generators, and INREP+ empirically outperforms others when the labeled data is scarce, as well as bringing more robustness against the label noise and the class imbalance.
In (A), we present full statements, proofs, and examples for all propositions, lemmas, and theorems in the main paper.
In (B), we present the details of the experiments' setting, including architectures, datasets, and evaluation metrics.

A Full Statements, Proofs, and Examples
A.1 Full statements and proofs for Section 2 A.1.1 Examples of the failure of ACGAN ACGAN has two objective terms L S , L C , modeling log-likelihoods of samples being from real data (by unconditional discriminator D : X → {real, fake}) and being from correct classes (by auxiliary classifier D aux : X → Y), respectively. Then, the discriminators (D, D aux ) and generator G maximize λL C + L S , λL C − L S respectively with a regularization coefficient λ. As in the vanilla GANs, L S term encourages G, D to learn the true (that is, unbiased) distribution, while L C term prefers easy-to-classify (that is, possibly biased) distributions. Therefore, G might be able to maximize λL C − L S by learning a biased distribution (increased L C ) at the cost of compromised generation quality (increased L S ). Indeed, this phenomenon was observed in [40], where the authors provided theoretical reasons and empirical evidence that ACGAN-generated data are biased. We corroborate their finding by providing two examples in which ACGAN provably learns a biased distribution. For simplicity, we consider the Wasserstein GAN version [14] of ACGAN (W-GAN). Lemma 1 (W-ACGAN fails to learn a correct Gaussian mixture, full statement). Consider 1-dimensional real data x ∼ N (y, 1) where y = 2 Bern(1/2) − 1. Assume the perfect discriminator and the following generator, parameterized by v: G(z, y) ∼ N (y, v 2 ), v ∈ R + for given label y, that is, the only model parameter is the standard deviation v. Let v * (λ) be the optimal generator's parameter that maximizes λL C − L S . Then, v * (λ) → 0 as λ → ∞.
Proof. Recall that L(v, λ) = λL C − L S for the generator. From the fact that the 2-Wasserstein distance between two Gaussian random variables is , the term L S reduces to a simple expression E y [W 2 (x, G(z, y))] = |1 − v|. Therefore, when there is no auxiliary discriminator, maximizing L(v, λ) is indeed the same as minimizing L S . Then, the ACGAN will find the global minimum v * = 1 via gradient descent as L S is convex. We will show that this is no longer true if we consider the auxiliary classification term L C .
Consider the L C term that also has a closed-form expression: where Q(·) is the complementary cumulative distribution function of the standard Gaussian, that is, Q(t) = ∞ t N (τ ; 0, 1)dτ . Therefore, the overall objective of the ACGAN to be minimized by G is To see the behavior of the best generator, that is, v * = v * (λ), first assume v < 1. Using the chain rule and the property of Q function that d dt As exp(− 1 2v 2 ) 1 v 2 is positive on (0, 1), the derivative is positive on (0, 1) as well by taking a large λ. Therefore, the generator will learn v * (λ) → 0 as λ → ∞. When v ≥ 1, So, the global minimum v * (λ) of L(v, λ) converges to 0 as λ tends to infinity. Fig. 7a shows the contour of L(v, λ) and demonstrates that v * (λ) converges to 0 as λ → ∞.
otherwise. An auxiliary classifier is linear that passes the origin such thatŷ(x) = sgn(tan(α)x 1 − x 2 ), α ∈ [0, 2π). The loss terms are: Note that L C term indicates classification error instead of accuracy. Let L( , α, λ) be L S + λL C . Assume the perfect discriminator. In this setting, L( , α, λ) has bad critical points, so gradient descent-based training algorithms may not converge to the global optimum.
Proof. Recall the Wasserstein distance between two 1-dimensional measures: where F −1 is the inverse of the cumulative distribution function of a random variable. First suppose 0 ≤ ≤ 1 so that the supports of p x|y (x|y), p G(z,y) (x|y) overlap. As our real and fake distributions are uniform, Suppose −0.9 ≤ < 0. Then, by the translation property of the Wasserstein distance, e.g., Remark 2.19 in [76], Therefore, the Wasserstein distance is given by The Wasserstein distance monotonically decreases in , so the vanilla (Wasserstein) CGAN finds * = 1 by gradient descent. In this example, d is chosen so that L S is continuous, and the mismatched range of is selected so that the errors of the auxiliary classifier on true data and fake data do not symmetrically move. We will see below that this solution could not be learnable by the ACGAN with gradient descent if one begins from a bad initial point.

A.1.2 Examples of the failure of ProjGAN
We provide an example to show why and when ProjGAN [2] may fail to work. Note that the ProjGAN's discriminator takes an inner product between a feature vector and the class embedded vector, that is, where y is an one-hot encoded label vector, is the label embedding matrix of y, and φ, ψ are learnable functions. This special algebraic operation reduces to expectation matching in a simple setting, by which we can provably show that ProjGAN mislearns the exact conditional distributions. Lemma 3 (full statement). Assume φ(x) = x and ψ(x) = 0 for an equiprobable two-class data. Thus, With hinge loss, the loss for the discriminator to minimize is written as Let us further suppose that the generator learned the exact conditional distributions at time t * , that is, G t * (z, y) ∼ x|y = y, where G t stands for the generator at time t. Then, there exist discriminator's bad embedding vectors that encourage the generator to deviate from the exact conditional distributions.
Proof. Note that the discriminator D under the setting is D(x, y; θ D ) = y T V φ(x) + ψ(φ(x)) = y T V x.
For simple presentation, we restrict ranges of x, v y properly so that D = y T V x is in (−1, 1). Then, we can omit the max operator. As classes are equally likely and our setting assumes D = y T V x, Hence, the gradient is in a simple form: which means that the discriminator measures the difference between conditional means of true data and generated data.
Note that the loss for the generator to minimize is . Its gradient is also in a simple form: Now we are ready to discuss why ProjGAN may fail to learn correct distributions. Let us suppose that the generator learned the exact conditional distributions at time t * , that is, G t * (z, y) ∼ x|y = y. This in turn implies E pdata(x|y) [x] = E z,y=y [G t * (z, y)] for y = 0, 1. Note that (4), (5) yield the following update: G t+1 (z, y) = G t (z, y) − η∇ G L G = G t (z, y) + α(v 0,t + v 1,t ) v y,t+1 = v y,t − η∇ vy L D = v y,t + α 2 E pdata(x|y) [x] − E z,y=y [G t (z, y)] .
Therefore, assuming that the step size α is small, alternating update yields the following dynamics after t * .
Repeating the updates, the G(z, y) steadily diverges from G t * unless v 0,t * + v 1,t * = 0. In other words, even when the generator exactly learns the true data at time t * , nonvanishing ∇ G L G may result in the divergence of the generator.