{'fraction_non_alphanumeric': 0.030389094007384265, 'fraction_numerical': 0.014381244997547185, 'mean_word_length': 4.935029114311983, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Conditional Generative Data-free Knowledge Distillation

Knowledge distillation has made remarkable achievements in model compression. However, most existing methods require the original training data, which is usually unavailable due to privacy and security issues. In this paper, we propose a conditional generative data-free knowledge distillation (CGDD) framework for training lightweight networks without any training data. This method realizes efficient knowledge distillation based on conditional image generation. Specifically, we treat the preset labels as ground truth to train a conditional generator in a semi-supervised manner. The trained generator can produce specified classes of training images. For training the student network, we force it to extract the knowledge hidden in teacher feature maps, which provide crucial cues for the learning process. Moreover, an adversarial training framework for promoting distillation performance is constructed by designing several loss functions. This framework helps the student model to explore larger data space. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments on different datasets. Compared with other data-free works, our work obtains state-of-the-art results on CIFAR100, Caltech101, and different versions of ImageNet datasets. The codes will be released.

Specifically, KD is the method to help the small student model training under the guidance of the teacher network. In deep models, the pre-trained teacher networks contain ample information about the original training data and target tasks, called knowledge. KD aims to transfer  Figure 1: The framework of the conditional generative data-free knowledge distillation method. In the adversarial training framework, the generator is guided to craft meaningful training data, and the student and teacher networks are combined as a discriminator. The student is optimized under the supervision of a teacher. the knowledge to student models and help them train. For the convolutional neural networks, Hinton et al. [9] first attempted to distill the knowledge to student networks by minimizing the discrepancy between network logits. Further works utilize the latent information in intermediate layers to improve the distillation performance. FitNets [10] utilizes the outputs (logits) and intermediate features together to trains the student models. Zagoruyko et al. [11] promoted student performance by forcing it to imitate teacher attention maps.
KD can efficiently compress cumbersome models when original or alternative data is available. However, training datasets for given models are often unavailable due to privacy and security limitations, such as federated learning tasks [17][18][19]. In addition, collecting alternative data is onerous, while irrelevant data will drastically deteriorate the performance of student networks. An effective way that tackles these questions is to reconstruct data as training samples. However, although the pre-trained teacher model contains rich knowledge of original training data, it is difficult to exploit the latent knowledge and reconstruct meaningful samples. Fortunately, generative adversarial networks (GANs [20]) offer a powerful capability to exploit data distribution and craft images. Recently, several data-free (zero-shot) knowledge distillation methods [21][22][23][24]that exploit generative adversarial networks have been proposed for compressing heavy models when real data is unavailable. And it can be employed on many extended tasks, such as the federated learning of medical data. However, due to the absence of conventional generation techniques such as GANs or VAE [25] cannot be directly applied to data-free KD. To this end, Chen et al. [22] trained a generator to create one-hot-like images and help the model distillation. These images will highly activate the neurons in teacher models. Fang et al. [23] presented an adversarial distillation mechanism to move generated samples toward the areas where the current student has not been well trained. Luo et al. [24] leveraged the intrinsic batch normalization statistics in the teacher model to guide the training of generators.
Although there are several data-free knowledge distillation methods, a performance gap still exists between the trained teacher and student models, while the data-driven KD can achieve even better performance than teacher models. In addition, the trained generator only produces class-balanced synthetic images that may be incompatible with some unique settings. Such as federated learning tasks which concern the class-imbalanced issue.
In this paper, a conditional generative data-free distillation framework is designed to attain efficient image generation and distillation, as illustrated in Fig.1. Specifically, we introduce preset labels as ground truth to build a semi-supervised conditional generator. With this supervision, we can adapt label distribution to match different datasets and avoid heterogeneity issues. Since preset labels will determine image classes, the student model can be optimized under the guidance of ground truth (preset labels). Besides distilling the outputs (i.e., logits), we also utilize the feature information (e.g., attention maps) that provides more precise and efficient guides to the learning process. Precisely, we force the student network to learn the feature information in the teacher model. It validates the importance of feature information and significantly improves distillation performance under data-free settings.
To demonstrate the efficacy of our method, we perform extensive experiments on several datasets and conduct a series of ablation experiments to analyze different designs. The main contributions of this work are summarized as follows: • We propose a conditional generative data-free distillation framework, which consistently produces meaningful training samples and attains efficient knowledge distillation.
• Our method trains a semi-supervised conditional generator by introducing preset labels as ground truth. The semi-supervised training process can address the class distribution issue by label sampling.
• We force the student model to learn teacher knowledge and consistently explore unseen data space in the adversarial training process. It can fully extract the feature information and improves student performance.
• Experiments on MNIST, CIFAR, Caltech101, and part of ImageNet datasets are conducted to demonstrate the superiority of our method. We obtain state-of-the-art on most datasets.

Related Work
In this section, we briefly review existing works on related topics.

Data-driven Knowledge Distillation
Knowledge distillation is a general model compression technique. In KD, we can transfer knowledge by minimizing network logit difference. However, teacher logits usually have a high probability in the correct class and close to zero in other classes. It cannot provide more information than the one-hot label. To this end, Hinton et al. [9] introduced SoftMax temperature to soften model output and extract more relationship information between different classes. Furthermore, Furlanello et al. [26] trained and integrated a set of student networks as the final student model. Mirzadeh et al. [27] introduced a middle-sized model (teacher assistant) as a bridge to help the student network training.
Besides distilling the logits, transferring knowledge to the student can be achieved by utilizing the feature information in hidden layers. Feature knowledge contains precise and efficient cues of the training process. In this case, Romero et al. [10] trained student networks using both logits and intermediate features. Zagoruyko et al. [11] transformed latent knowledge into attention maps and forced student networks to mimic the attention maps of the teacher. Yim et al. [28] introduced the FSP (Flow of Solution Procedure) matrix to learn the relationship between inputs and outputs. Park et al. [29] proposed a relational knowledge distillation method. They transferred the mutual relations of data according to the distance-wise loss and angle-wise loss. Bai et al. [30] designed a novel layer-wise cross distillation method and realize few-shot distillation. Li et al. [31] trained a portable objective detection model based on the supervision from proposal features. Liu et al. [32] presented a dense prediction distillation method based on holistic knowledge and pairwise similarity. Some recent works [33,34] also explored the structure effect to model distillation. Moreover, combining the knowledge distillation with other techniques (e.g., NAS [34], RL [35], or GNN [36]) can further compress and accelerate the heavy models. For other related works, [37] reviewed and looked at the development of knowledge distillation.
Although these data-driven KD methods have achieved outstanding performance in many applications, there are many challenges when real data is unavailable. Exploring more efficient knowledge representation and utilization techniques for data-free tasks is necessary.
In data-free KD, Lopes et al. [21] attempted to reconstruct original data from metadata and utilized the synthetic data for training student networks. However, it still requires metadata extracted from real data. Nayak et al. [43] modeled the SoftMax space as a Dirichlet distribution and updated random noise images to craft Data Impressions. Chen et al. [22] designed a novel framework (DAFL) to train a generator that matches the original data distribution. They fixed the teacher model as a discriminator and designed several novel loss functions to optimize the generator. In contrast, Fang et al. [23] presented an adversarial distillation mechanism (DFAD). In DFAD, teacher and student jointly play the role of a discriminator to reduce prediction discrepancy, while the generator adversarially enlarges the discrepancy by creating hard samples. Yin et al. [44] synthesized images through model inversion and needed no additional generators. They utilized the information from batch normalization statistics to improve synthetic images. For data-free distillation on large-scale datasets, Luo et al. [24] trained one generator for each class. It gains remarkable performance and avoids mode collapse (the generator produces similar images and loses diversity) but requires vast computational resources. Meanwhile, Jin et al. [45] used a simple fully connected network as an additional discriminator to distinguish the teacher and student models. Fang et al. [46] improved image diversity by introducing contrastive learning.
These methods have realized data-free distillation, but the feature information which provides crucial cues to the learning process is not fully used. In this case, the student model is unable to obtain desirable performance. Moreover, the generator can use the class information and tackle heterogeneity issues through label shift. In this work, we propose a more efficient approach to attain data-free image generation and distillation.

Generative Adversarial Networks
To transfer knowledge from the teacher to the student network when training data is inaccessible, we adopt the framework of GAN to produce training samples. GAN contains two interacting neural networks: a generator G that captures data distribution and a discriminator D to distinguish the real and fake samples (generated by G). The two networks are trained jointly through a minimax game. In this adversarial process, G learns to generate meaningful samples that the match original distribution. To make the generator produce specified classes of images, Mirza et al. [47] proposed the conditional GAN. It applies class conditions to both generator and discriminator networks. Furthermore, Arjovsky et al. [48] presented Wasserstein GAN (WGAN) to overcome training instability and mode collapse issues. Similarly, Qi et al. [49] regularized the adversarial loss through the Lipschitz regularization.
In recent years, GAN has achieved great success in many image synthesis areas [50]. Some works attempted to improve computer vision tasks through the GAN module. To help the training of the fine-grained human action segmentation mode, Gammulle et al. [51] developed a recurrent semi-supervised action GAN, which learns an intermediate representation. Zheng et al. [52] enhanced the person re-identification (re-ID) model based on a generative module. The generative module learns to produce better images for conducting teacher-student distillation, while the re-id model (discriminative module) mines and extracts better subtle features. Zhang et al. [53] designed a semi-supervised GAN for precise automatic defect detection that can utilize unlabeled data to improve segmentation results. These works apply the GAN module to enhance different deep models in a semi-supervised learning manner. Theoretically, the powerful GAN module can be applied for data-free KD. However, GAN-based methods aim to craft realistic images and require real training data. Therefore, developing an efficient data-free GAN framework is necessary when conventional GAN fails in data-free tasks. We train a semi-supervised conditional generator to produce training data by exploring the hidden original data distribution in the teacher model.

Method
This section describes the proposed conditional generative data-free distillation framework. As shown in Fig.1, the framework contains three networks: a pre-trained teacher network T , a smaller student network S , and a generator G. During training, a set of random noise z = z 1 , z 2 , ..., z n is inputted into G to produce training samples G(z), where n is the number of samples, z i ∈ R m is sampled from the normal distribution p z .

Generative Loss Design
In previous data-free works, the trained generators only produced images randomly. It has no clear optimization objective for the generator and classifiers (i.e., teacher and student models). However, class information is crucial for distillation and data generation, so we introduce preset labels as ground truth and build a conditional generator in this work. Under the supervision of preset labels, the trained generator can produce images of specified classes and meet different requirements (e.g., class-imbalanced datasets). To be specific, we uniformly sample a set of preset labels y i from {0, 1, ..., c − 1} as conditional information, where c is the number of classes. The class matching loss can be defined as: where y i is the preset label of i-th image, and H(·) denotes cross-entropy loss. l i T = T G z i | y i is the logits of teacher network. G(z | y) is the synthetic image of class y. This loss function calculates the cross-entropy between teacher logits and preset labels. In order to guide the generator training, we hope the teacher model can produce the same predictions as preset labels. For class-imbalanced tasks, we will sample preset labels according to label distribution. With the class matching loss, the generator has a clear optimization objective. And we convert the generator training into a semi-supervised learning process.
In practice, we do not only optimize L CM when training the generator. To measure the class balance of generated images, we introduce information entropy loss and formulate it as: wherep i = 1 n j l j T is the average logit at class i, H in f o (·) is information entropy. L ie forces the generator to produce a set of balanced samples, and it takes the minimum whenp i equals 1/c. Actually, this loss complements the class matching loss in Eq.1 and can be utilized together to promote the image class balance. Noting that, the information entropy loss will be adjusted for class-imbalanced tasks.
Besides information entropy, teacher output is critical information of the training data. In classification tasks, high-performance models produce one-hot-like vectors for input images. If synthetic samples follow the same distribution as original data, teacher models will also output one-hot-like vectors. In other words, the input image is matching original data distribution when the teacher outputs a one-hot-like vector. To this end, we compute the cross-entropy between predicted labels and teacher logits and formulate the one-hot loss as: is the prediction of the teacher model. With the supervision of onehot loss, we guide the generator to craft images that match the original data contribution. The one-hot loss can also be used with the class matching loss: L CM requires the synthetic image to be a specified class, while L ie enables the images to have more prominent features of a specific class. Since information entropy loss and one-hot loss have no clear optimization objective, we combine them as the unsupervised loss: where λ ie is scale weight for balancing different components. Loss functions in Eq.1 and Eq.4 utilize the knowledge from the teacher model. In datafree distillation, however, the student network is trained together with the generator. Naturally, student information is beneficial for training a generator compatible with the student. So we introduce discrepancy estimation loss to measure the gap between student and teacher predicted results. In this case, the information from the student can be used to train the generator. Moreover, KLD (Kullback-Leibler Divergence) and MSE (Mean Square Error) lead to gradient decay [22] when the student model converges on synthetic images. It will deactivate the generator learning and make the minimax game fails. To tackle this problem, Mean Absolute Error (MAE) is adopted to compute the discrepancy estimation loss: where T (G (z | y)) and S (G (z | y)) are the outputs of teacher and student networks. The student and teacher models should produce the same predictions for synthetic images to minimize L DE in the training process. However, this function will induce the generator to craft easy samples. With these easy samples, the student network will quickly learn to produce the same prediction as the teacher model, but it only learns shallow knowledge. Actually, deep latent knowledge can be transferred by exploring hard samples: we conversely maximize this loss to move the generator towards the areas where the current student network is not well trained.
Combining the loss functions mentioned in Eq.1, 4 and 5, we define the generative loss as: where λ US and λ CM are scale weights to trade off different loss terms. In the generation stage, we train the proposed conditional generator to craft training samples by minimizing the generation loss. Besides, we utilize batch-norm statistics to promote the training process further.

Distillation based on Generated Images
In section 3.1, discrepancy estimation loss is employed to measure outputs discrepancy of the teacher and student networks. We maximize this loss and induce the generator to produce hard samples. In the distillation process, we reversely minimize this loss function by forcing the student network to imitate teacher output. Thus, we construct an adversarial learning framework. The generator consistently produces hard samples and enables the student network to explore unseen regions.
Besides estimating discrepancy, ground truth is also beneficial for distillation and is commonly used in data-driven KD. Although ground truth in data-free settings is unavailable. We utilize the information from preset labels in Eq.1 to facilitate distillation and narrow the data-free and data-driven KD gap. The ground truth loss is defined as: where l i S = S G z i | y i is the logits of the student network. Different from L CM , we calculate the cross-entropy loss between student predictions and the preset labels. In this situation, the student model has a clear optimization objective.
Although L DE and L GT provide helpful information, they are insufficient for complex tasks. The student model cannot learn valuable knowledge if there is a huge capacity gap between the teacher and student networks, especially in data-free settings. Fortunately, intermediate features offer crucial cues to the learning process. In the natural world, attention plays a critical role in human visual experience. Zagoruyko [11] claimed that attention maps in CNNs show the area where the network is concerned. In other words, the pixel with a larger attention value has a higher probability of belonging to the object. We extract the feature knowledge hidden in intermediate layers by introducing the attention transfer loss and formulate the activation-based attention as: where A denotes activation in each channel, and C is the number of channels. Power and absolute value operations are element-wise. Then the attention transfer loss can be computed as follows: in which Q j S = vec F A j S and Q j T = vec F A j T are the j-th pair attention maps of the student and teacher networks. I is the number of layers that perform attention transfer operations, and vec(·) converts the feature maps into vector form. We guide the student to mimic teacher attention maps by minimizing L AT .
Combining the loss functions in Eq.5, 7 and 9, we yield the distillation loss: where λ GT and λ AT are hyper-parameters to balance different components. Based on the distillation loss L KD , the student network learns to extract features and make predictions. It is noted that previous work did not fully use teacher knowledge, consequently leading to an accuracy gap. In contrast, our method gets better performance through efficient information extraction.

Algorithm 1 Conditional Generative Data-free Knowledge Distillation Framework
Input: A given pre-trained teacher network T , a normal distribution p z and hyper-parameters for balancing different terms Output: Student network S and generator G 1: Initialize the parameters in student network S and generator G 2: while not converged do Generate training samples x → G(z | y); 6: Calculate the loss of generator L G in Eq.6; 7: Update the parameters in generator G according to the gradient; 8: Train the Student Network 9: for k iterations do Compute distillation loss L KD in Eq.10; 12: Update the student network S using back-propagation; 13: end for 14: end while The conditional generative data-free knowledge distillation framework for learning efficient portable networks is summarized in Algorithm 1. The training procedure is divided into the generation and distillation stages. We alternately update the parameters in student S and generator G, while the teacher model T is fixed.
In the generation stage, we sample a set of preset labels y i n i=1 and noises z i n i=1 to craft training images G(z | y). These images will have the same label distribution as the source 8 dataset. G learns to produce meaningful samples with a similar distribution to the original data by minimizing L G in Eq.6. Since minimizing L G will increase L DE , the generator tends to produce hard samples. In the distillation stage, the generated images are employed as training data and force S to explore unseen data space. By minimizing L KD in Eq.10, we guide S to learn the knowledge from T . The student network not only learns teacher output but also mimics attention maps. Unlike the generator training, we update the student network k times each epoch to ensure it converges. Due to L DE decreasing with L KD while increasing with L G , we construct an adversarial process between the generation and distillation process. In the whole adversarial training process, G continuously produces hard samples, and S consistently learns valuable knowledge from T . We repeat the two training processes alternately until both models converge.

Experiments
In this section, we conduct extensive experiments on different datasets to validate the effectiveness of the proposed method. Furthermore, a series of ablation experiments are performed to analyze different components in this work.

Datasets and Models
We conduct experiments on several datasets, including MNIST, CIFAR, Caltech101, and part of ImageNet. MNIST. MNIST [54] is a well-known image dataset of handwritten digits composed of 28×28 pixels gray-level images. It is very simple and only contains ten classes (from 0 to 9) with 60,000 training and 10,000 testing images. We adopt LeNet-5 [55] and LeNet-5-Half [21] as the teacher and student networks. CIFAR. CIFAR10 and CIFAR100 [56] are more complex image datasets than MNIST. CIFAR10 consists of 32×32 pixels color images in 10 classes, while CIFAR100 has 100 classes. They both contain 50,000 training samples and 10,000 testing samples. Caltech101. Caltech101 dataset [57] consists of 101 classes. Each class has 40 to 800 images. In the training process, we randomly split 80% images as the training set and 20% images as the testing set. These images are larger than 128×128 pixels. ImageNet. ImageNet1k [58] dataset provides more than 1.28 million training and 50,000 validation samples in 1000 classes. Due to the complexity of the dataset and the limitation of computational resources, we decompose this task into several subtasks on different datasets: Im-ageNet100 and ImgaeNet32. ImageNet100 consists of 100 randomly selected classes from the original ImageNet dataset. ImageNet32 is a down-sample version of ImageNet. It includes all images of ImageNet, resized to 32×32 pixels.
ResNet34 and ResNet18 [59] are adopted as the teacher and student networks for all datasets except MNIST.

Evaluation Metrics
Prediction accuracy is often used to evaluate different classification algorithms, while the accuracy gap between teacher and student models is employed in distillation works. However, teacher models in each work have different accuracy, and it is not easy to compare the effectiveness of different distillation methods. In this paper, we propose an additional evaluation metric and define it as relative accuracy: where T acc and S acc are the accuracies of teacher and student networks. With relative accuracy, we can directly compare the effectiveness of different distillation methods.

Talent Selection Strategy
If the effectiveness of methods or hyper-parameters has not been confirmed, completing training will waste resources. We propose a simple talent selection strategy to reduce training time and computational cost. The strategy is a manual tuning technique that helps us verify new methods and adjust hyper-parameters with minimal consumption. For example, if the model requires 200 training epochs and adjusts the learning rates twice, we train it for 50 epochs or less and change the learning rates at the 40th epoch. After several selection experiments, we can verify current settings by comparing the results and trends. Then we choose 2-4 group better settings for complete training. Although the final results may not be globally optimal, it is a balance between network performance and computational cost.

Experimental results
We implement all experiments with PyTorch and do not use any data augmentation technique for a fair comparison. For all datasets, SGD with momentum 0.9 and weight decay 5 × 10 −4 is applied to update the student networks, and Adam is employed to optimize the generators. We use the generators following [23] and make adjustments to match our method.

Experiments on MNIST
We first implement the method on MNIST. The images are resized to 32×32 pixels in the training process, and the batch size is 512. The initial learning rates are 0.01 and 0.001 for student and generator, respectively. For MNIST, we train the models 60 epochs and decay learning rates at the 50th epoch. Each epoch iterates 50 steps, k equals 5. In this experiment, the student network will quickly converge to high accuracy. For simple tasks such as MNIST, we should focus on the training epoch and avoid the model over-fitting.
Table1 reports the results of different data-free distillation methods on MNIST. With the absence of real training data, DAFL [22], DFAD [23] and DFKD [45] obtained 99.28%, 99.32% and 99.53% relative accuracy, respectively. After completing adversarial training, the student network in our method achieves better results of 98.62% prediction accuracy and 99.65% relative accuracy. We get a little but non-negligible gain when accuracy is very close to the upper bound. It demonstrates that our method can explore larger data space.

Experiments on CIFAR
For CIFAR datasets, the initial learning rate of student networks and generators are 0.1 and 0.001, respectively. We use the training epoch of 250 and decay the learning rates twice. In each epoch, we iterate the training process 100 steps and use the batch size of 256. Under this setting, the student model sees 256×100×5 (k=5) synthetic images in each epoch. It is even more than the images in the original CIFAR datasets. In this case, student models can sufficiently explore

Method
T acc (%) S acc (%) Rel acc (%) DAFL [22] 98  the data space and get good results. Since CIFAR is more complex than MNIST and will take a lot of training time, we adopt talent selection strategy to tune the hyper-parameters. After the selection process, three better settings were chosen for complete training. Table 2 summarizes the experimental results of different works on CIFAR. For CIFAR10, Google presented LS-GDFD [24] and achieved the best results of 99.97% relative accuracy. It only has a 0.03% prediction accuracy decay. We achieve comparable results despite not having as many computational resources as Google (they used the batch size of 32678, and ours is 256). The trained student network in our paper yields 99.63% relative accuracy and 95.19% prediction accuracy on CIFAR10. It is only slightly worse than LS-GDFD and far outperformed other methods. For CIFAR100, which is more complex than CIFAR10, we obtain the best results of 76.80% student accuracy and 99.07% relative accuracy. It shows that our method is not only applicable for simple tasks but also for complex datasets. It is worth noting that LS-GDFD trains one generator for each category of images to avoid mode collapse and improve results. We only compare the results with those obtained on a single generator for a fair comparison. Fig.2 (left and middle) shows the synthetic images on CIFAR datasets. We can find that data-free generators tend to produce semantic images due to the lack of real images, while the conventional GAN produces realistic images.
Furthermore, we multiply the random noise and label embedding rather than concatenate them. We find that the generator will easily produce images with the prominent class feature when adopting the concatenate operation. However, it does not achieve good results. In contrast, the generator with the multiply operation tends to produce weird images but leads to better distillation performance. We think the multiply operation will craft hard samples, while hard samples Figure 2: Synthetic images of CIFAR10 (left), CIFAR100 (middle) and ImageNet32(right), each column has the same category. Top: real images sampled from real datasets. Bottom: images generated by the conditional generator. Notably, the generator tends to produce semantic rather than natural images because there are no real images as the learning target.
will induce the student model to explore unseen data space and get better performance.

Experiments on Caltech101
We extend experiments to other high-resolution image data to further validate our method. In the training process, we resize the images 128×128. Similar to the experiments on CIFAR, the talent selection strategy is applied to Caltech101. In order to generate larger images, we choose the DCGAN [61] and make some adjustments to match our method. The generator in DCGAN is generally used in data-free distillation works and will provide a fair comparison. Table 3 reports the experimental results of Caltech101. While DFAD and DDAD obtained 73.5% and 75.01% prediction accuracy, our method gets 76.46% student accuracy and 99.84% relative accuracy. We get state-of-the-art results and a 1.9% relative accuracy gain. It shows that our method can be extended to tasks with larger images. To validate the efficacy of feature information, we visualize the attention maps of teacher and student models in Fig.3. Different colors indicate how much attention it receives, while the magenta region receives the most attention. The visualization results show the student model can well notice the local and global features of images. In other words, the student has well learned the knowledge from the teacher model.

Experiments on ImageNet
ImageNet is the most well-known image classification dataset, providing 1000 classes of images with a resolution greater than 224. However, as far as we know, only LS-GDFD implements data-free distillation on this task. Based on the ample computing resources of Google, they train 12  1000 generators for each class. It is helpful but requires heavy computational resources. To this end, we decompose this task into several subtasks on two datasets in this work. Table 4 summarises the results on ImageNet32 and ImagNet100. For ImageNet32, we validate the distillation performance for more classes. The images in the ImageNet dataset are more diverse than in other datasets. Train one generator for all classes will inevitably meet the model collapse. Even so, our method gets 50.48% relative accuracy with only one generator, which exceeds the results of LS-FDGD obtained on 100 generators. Our method will perform better under the same settings since multi-generators can efficiently avoid model collapse. Fig.2 (right) shows the generated images of ImageNet32. Then we randomly sample 100 classes from the original ImageNet dataset to make up the ImageNet100 dataset and conduct experiments. We resize the images to 224×224 and verify the efficacy of generating larger images. For complex generation tasks (e.g., the images in original ImageNet are more diversity), some special designs ensure efficient image generation (e.g., BigGAN [63] and SAGAN [64]). These designs are beneficial for this work but lead to unfair comparison. Some previous attempts [65,66] claimed that high data diversity worsens image generation. While not using any special designs, we achieve an acceptable result of 71% relative accuracy on the size of 224 × 224. Moreover, the distillation performance will get degraded when the capacity gap between the teacher and student models is significant [37]. In the future, we will explore this area to reduce performance degradation.
In the above sections, we perform extensive experiments on different tasks with image sizes ranging from 32 to 224 and image classes ranging from 10 to 1000. The experimental results demonstrate that our method can be well applied to most datasets.

Ablation Study
In the above sections, we have conducted extensive experiments on different datasets and demonstrated the effectiveness of our method. However, our framework contains several loss components that play different roles. We will analyze the following novel loss functions, including ground truth loss (L GT ), class matching loss (L CM ) and attention transfer loss (L AT ). In this work, we make the preset labels supervise the student model by utilizing L GT , and it only works when we use L CM to control the image class. To explore these functions, we perform a series of ablation experiments on CIFAR100 following the experimental settings in section 4.3.2. The results of various design components are summarized in Table 5.
Student accuracy without any novel loss function is 74.81%. We first introduce the attention transfer loss to this framework, improving the student accuracy to 76.34%. We get a 1.53% performance gain on CIFAR100. It suggests that the attention transfer can remarkably improve distillation performance. Actually, a trained teacher model will focus on the target area of input images and capture valuable information. With the guidance of attention transfer, student network will learn the knowledge from teacher and achieve better performance.
Many feature-based distillation works presented efficient information extraction methods except for transferring the attention maps. Although these data-driven methods improved distillation performance, the challenges remain in data-free tasks. In practice, we can represent hidden information in many forms, and the student model can well learn the knowledge by exploring training data. However, the knowledge with some forms may not be easy to learn, especially when the training data is not accessible. To this end, more effective knowledge representation methods should be explored to attain efficient knowledge transfer.
In the following experiments, we find L CM also prominently improves student accuracy and gets a 0.7% accuracy gain. It is because the generator will converge better when subjected to additional constraints. We obtain 76.71% prediction accuracy when combining L CM with L AT and 75.56% accuracy when combining L CM with L GT . Although the ground truth loss does not lead to remarkable improvements, it greatly accelerates the model converges. When using all of these components, our method gets the best performance of 76.80%. The ablation experiments demonstrate the effectiveness of each component in our work. In addition, the generator produces meaningless images in the early stage, so the unsupervised loss L UN is necessary for stable training.

Conclusion
In this work, we present a conditional generative data-free distillation framework. It trains lightweight student networks through efficient image generation and knowledge distillation. This framework can be divided into two adversarial processes. We treat preset labels as ground truth in the generation process to train a conditional generator. Semi-supervised training can avoid the class-imbalanced issue in heterogeneity tasks through label sampling. In the distillation process, the generator produces hard samples as training data. We force the student model to fit these hard samples, thus learning teacher knowledge and exploring unseen data space. In conclusion, we construct a whole adversarial learning framework base on the data generation and exploring processes. Experimental results show that the proposed method remarkably improves student performance and yields state-of-the-art results on most datasets. Furthermore, this method can be well applied to extended data-free tasks, such as federated learning on medical images.

Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.