{'fraction_non_alphanumeric': 0.026098113619611805, 'fraction_numerical': 0.006661024815047364, 'mean_word_length': 5.023018743834265, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Knowledge Within: Methods for Data-Free Model Compression

Background: Recently, an extensive amount of research has been focused on compressing and accelerating Deep Neural Networks (DNN). So far, high compression rate algorithms require part of the training dataset for a low precision calibration, or a fine-tuning process. However, this requirement is unacceptable when the data is unavailable or contains sensitive information, as in medical and biometric use-cases. Contributions: We present three methods for generating synthetic samples from trained models. Then, we demonstrate how these samples can be used to calibrate and fine-tune quantized models without using any real data in the process. Our best performing method has a negligible accuracy degradation compared to the original training set. This method, which leverages intrinsic batch normalization layers' statistics of the trained model, can be used to evaluate data similarity. Our approach opens a path towards genuine data-free model compression, alleviating the need for training data during model deployment.


Introduction
Quantization is a prevalent, accelerator friendly, compression method [13,11] employed prior to DNNs deployment within real world applications. However, high com-pression rates typically demand additional information to minimize quality loss. For example, when using low precision arithmetic, it is often beneficial to trade-off numerical representation range with resolution by clipping the dynamic range based on its real expected values. Thus, it is a common practice to gather per tensor statistics [13,1] from a subset of samples drawn from the training set (Calibration). Furthermore, in cases of very high compression rates, model parameters often require additional adjustment to recover from catastrophic error accumulation, where access to the entire training data is needed. This may lead to an undesired coupling between deployment and training phases of a model through data. Especially in cases where the training data is sensitive or simply unavailable at time of deployment. Therefore, it is appealing to investigate new methods to alleviate the need for real data for deployment purposes, for example, via substitution with synthetic data.
Generating high quality samples requires capturing the prior distribution of the data which is often hard. A large body of work dedicated to generative models has shown that it is possible to learn such priors and generate high resolution synthetic images [5,14,25,8,4]. Unfortunately, these techniques comes at the cost of training a dedicated generative model which requires access to the real training data which we aim to avoid.
In this work, we aim to understand the potential and limitations of synthetic samples for model compression tasks, specifically for reduced precision deployment i.e. calibration and fine-tune via Knowledge Distillation (KD [9]). In the context of this work, the full precision model (or parts of it) serves as the teacher, while the student is the low precision counterpart. We will focus our attention on DNNs for classification tasks on Cifar and Ima-geNet [16,24]. In the process, we revisit a feature visualization process dubbed "Inceptionism" [20] which is based solely on the trained model. This process typically starts with an arbitrary input, which is iteratively adjusted to maximize the response of a set of target features via back-propagation. Furthermore, the optimization process is typically constrained using some prior knowledge about the input, such as high correlation between nearby pixels within an image to avoid sample over-fit. Similar approach was recently adapted for several use-cases including for the purpose of data free distillation [21,3,17] with limited success.
Our contribution is twofold: First, we offer a novel methods for generating and leveraging synthetic samples for the use of knowledge distillation. These samples are created under a realistic data-free regime by exploiting the encapsulated knowledge within the provided model. We then empirically evaluate the usefulness of these samples for model compression, yielding comparable results to the original training dataset with minimal accuracy degradation. Second, we propose a novel approach for evaluating similarity between a reference dataset and a set of arbitrary samples. In a nutshell, we suggest measuring the mean divergence with respect to low order statistics drawn from a set of intermediate layers of a given model which was trained on the reference dataset (reference model). This can be done without relaying on an access to any real data by leveraging intrinsic measurements provided by Batch Normalization (BN) layers [12].

Related work
The notion of "data-free" quantization was recently introduced by Nagel et al., [21]. This is achieved by using the pre-determined measurements of batch-normalization statistics [12], that are gathered during training to determine the proper dynamic range for each layer. However, this method requires either access to all layers output statistics, or relaying on a closed form analytical solution for missing layers statistics based on their input distribution. In addition, this method is unable to handle cases of extreme degradation where fine-tuning is required.
Previous work explored using data-free knowledge distillation for model compression by collecting metadata related to the statistics of a trained model output [17,3]. In both cases, the generation scheme consists of sampling a set of random target tensors and minimizing the mean square error between the output and the sampled targets. Lopes et al., [17] proposed collecting means and co-variance for a set of layers after training, while optimization targets are sampled per layer independently under multivariate Gaussian assumption. The authors mention this method fails to capture inter-layer relations and proposed an alternative approach to capture inter-layer relations via graph spectral analysis with compelling results for models trained on MNIST dataset. However, this is impractical for large models or models with a large input size, due to its computational cost.
Bhardwaj et al., [3] suggested it is sufficient to collect metadata from the layer prior to the linear classification. Metadata collection is done by processing a small portion of the training set, clustering the high dimensional outputs and applying Principle Component Analysis (PCA) per cluster. The optimization target is based on the collected centroids and a random noise projected in the direction of the primary principle components. Bhardwaj et al., [3] presented a small set of experiments on Cifar10 dataset, with relatively large degradation between real data to the generated samples.
Recently, Nayak et al., [22] proposed similar data free method dubbed as zero shot distillation, which relies solely on the final layer weights to compute a class similarity matrix. Then, sample soft targets via Dirichlet distribution for generating synthetic images. However, KD results on synthetic samples are only comparable to real data on easily separable datasets such as MNIST, while performing poorly on Cifar10.
In this work we aim to generate samples which mimic the training data distribution by some measure. This can be achieved with a naive noise sampling according the low-order statistics of original data, or directly optimizing the similarity measure of the internal statistics in a given pretrained model. In contrast to previous attempts [17,3] which required sampling activation generated by real data, our method uses the recorded knowledge on low order statistics captured by BN layers (i.e., channel-wise mean and standard deviation). We then track the statistics induced by the synthetic samples and directly optimize the divergence with respect to the reference set while naturally relying the model structure to maintain inter-layer statistics' relations.

Methods for data-free distillation
We are interested in a data-free regime, wherein a model is given without its corresponding dataset used for training. This regime reflects a realistic scenario, as training data is often confidential or private. Therefore, we offer three alternative methods for generating useful synthetic data for distillation and calibration: Gaussian Scheme: The model's inputs are generated at random by sampling from a Gaussian distribution. Inception Scheme: The data is generated via activation-maximization (a special case of the Inceptionism scheme). BN-Statistics Scheme: The data is generated by optimizing a novel internal statistics' divergence measure.

Gaussian Scheme
As a naive approximation of the original dataset, it is natural to consider a simple Gaussian generator (denoted as G) with first and second moments defined to match the original input data. We suggest this alone may be sufficient for model calibration task required for low numerical precision inference, under mild compression demands. Such generation scheme is appealing since indefinite number of samples can be generated at will with minimal compute and storage requirements. However, as one would expect, under extreme compression demands, when the model parameters need to be adjusted (e.g., fine-tune using distillation), this method may prove to be insufficient.
Since this method does not preserve the original input's structure, the internal activation's statistics may differ significantly from the original statistic induced by the training data. This change can harm the model accuracy especially if it contains Batch-Normalization (BN) layers [12]. BN layers are commonly used in DNNs workloads, as they have been shown to improve both accuracy and speed of convergence by normalizing the input to the layer to have zero mean and unit variance before applying its operation. During training each BN layer keeps a running estimate of the empirical mean and standard-deviation per-channel of its inputs which latter applied to normalized the input data during inference.
Adjusting the model's parameters by using distillation over the randomly generated samples, will irretrievably alter the existing BN layer parameter to accommodate for the observed statistical properties, resulting in an imminent failure when turning back to evaluate real data. Thus, we suggest forcing all BN layer in the model to maintain their original running estimates for evaluation. As shown in section 4 this may negate this obstacle to some extent, enabling the use of such samples for the task of distillation.

Inception Scheme
Inceptionism [20] related generation schemes, typically impose constraints solely on the input and output of the model. We shall focus on a special case, which we term Inception-Scheme (denoted as I), where only a single neuron from the final model output is maximized under an input smoothness requirement. We define the optimization objective as the sum of a Domain Prior term and an Inception Loss term. As a domain prior for images, we use a Gaussian smoothing kernel to produce a smoothed version of the given input. The loss term is then computed as the mean squared error between the input image and the smoothed variant, encouraging nearby pixels to have similar values. The Inception Loss term, is derived by choosing an arbitrary label and perform gradient descent on the negative exponent of the appropriate class logit drawn from the Fully Connected (FC) layer output, i.e. e −logit/scale . The Inception Loss injects the desired class information, where the exponent and temperature-scale control the impact of the logit magnitude on the loss, preventing the model from producing inputs that cause the output to explode by exponentially decaying the logit contribution to the general loss as its magnitude increases. This scheme often results in high confidence outputs, however, it is highly sensitive to the hyper-parameters (detailed experiments are given in Appendix section B).

BN-Statistics Scheme
Based on our Gaussian and Inception scheme experiments we hypothesize that the lack of regularization on the in-ternal statistic during sample generation process may result in significant internal statistics' divergence with respect to the observed statistics on real data. Indicating such samples are not drawn from distribution similar to the original data (as we show later, in Figure 1a. This in turn, may impede their use for model compression and KD. Thus, our proposed approach is to directly minimize the internal statistics' divergence, by optimizing a novel measure which we term "BN-Stats" (denoted as BN S). BN S only assumes access to the predetermined empirical measurements that occur for each BN layer. We will use these estimates as targets and compare new data samples by the similarity between their measured statistics.
More formally, given the running estimatesμ andσ of a given batch-norm layer, we wish to measureμ = µ(D) andσ = σ(D) of our new data source D and evaluate their similarity to the reference statistics. BN S define this similarity using the Kullback-Leibler (KL) divergence under a simplistic isotropic Gaussian assumption such that Our generation process starts with a batch of random input samples, which are then iterativly adjusted to minimize the mean statistical divergence across all layers in the reference set. Namely, in each optimization step over an input batch X, we extract per-BN-layer activation set {D l } N l=1 with N BN-layers matching the reference BNlayers set, {µ l , σ l } N l=1 . We then compute the mean over all BN S(D l , µ l , σ l ) as the optimization objective, followed by standard backpropagation to adjust X. Thus, given a set of measurements from original training data {µ l , σ l } N l=1 and the matching measurements {μ l ,σ l } N l=1 for input batch X the target and source distribution sets under the isotropic Gaussian assumption are: and the optimization objective (i.e., BNS measure) is the mean KL divergence on the set {T, S}, i.e.
Finally, we add small constant, = 1e −8 , to the measured variance of the induced synthetic distribution to accommodate for zero-variance channels. We note that alternative metrics can be used for BN S under the same underlying assumptions, such as Mean Square Error over the empirical moments.

Combining BN S + I
In addition to the three schemes described above, we experiment with a combination of BN-Statistics and Inception schemes objectives denoted as BN S + I, see Algorithm 1. This method of generation attempts to harness the strength of each method. The class constraint imposed by I impacts the induced statistics depending on the composition of the batch, forcing the optimization process to compensate for randomly sampled classes which is more similar to the statistics observed during real data training. The drawback of this method is the additional loss scaling parameters which are now added to the original hyper-parameters of the Inception scheme. We didn't exhaustively investigate the best method to combine the two however, we present results using a simple aggregation which in some cases provides an encouraging improvement.

Generating data samples
We first describe the shared setting for sample generation, used by the methods introduced in section 3. As the generation process must start with a previously trained model, ResNet architectures is used for our experiments as they are a popular examples for networks in which training quality heavily relies on batch normalization layers. We continue to describe the components and implementation details for sample generation: Algorithm 1: Generating BN S + I samples Input: a pre-trained model with N BN layers Param: input shape, batch size, budget Param: scale, k smooth , σ smooth , α, β, γ Param: optimizer(θ), #duplicates Output: a batch of synthetic samples -X budget 1 Init: X 0 = random init(batch size, input shape) 2 Init: targets = randint(batch size, #classes) When enforcing onerous compression rates on a model it is often required to retrain the model under the compression constraints to recover from any significant accuracy loss. The process of adjusting the model parameters to the compression constrains is commonly called fine-tuning.
Since each dataset/model responds differently to quantization, we choose numerical precision to ensure that calibration or fine-tuning is needed. We will consistently use the notation of XwXa, denoting the number of numerical precision bits used for representing the quantized activations and weights throughout the model. In all our experiments the case of uniform quantization [27] is considered, with per-channel scale for the weights and a per-tensor scale for the layer's activation values. A copy of the weights [11] as well as the biases and gradients are kept in single precision where the latter is derived using straight through estimator [2]. All results are reported for simulated quantization (i.e. discretizing the inputs and weights before applying float operators). Ad-ditionally Batch-Normalization layers were kept in single precision.
Calibration details. Our experiments begins by calibrating each model using real and synthetic data, under increasing compression rates. We then perform KD by setting the float model as the teacher and apply the most demanding compression settings to the student model. During calibration, we use a smoothed absolute dynamic range measurement. That is, the dynamic range is measured via running estimates of the mean absolute min/max values within chunks containing 16 samples each. Unless otherwise specified, 200 calibration steps are used with batch size 256. Samples are drawn with replacement out of a balanced dataset with size limited to 1% of the training dataset size. Standard data augmentations are applied on each inputs batch to maximize available data utilization (e.g., random crop and mirroring). We argue that more advanced calibration methods [21,1] may improve results as they are uncoupled to the proposed data generation schemes.
Fine-tuning details. In all our fine-tuning experiment we use classic KD with teacher labels only. Optimization settings were fixed throughout each set of experiments to enable fair comparison. We begin with calibrating the model on the training set, and follow McKinstry et al., [18] by freezing the student's measured dynamic ranges for each layer's activation for the rest of the optimization process (we didn't find the suggestion of freezing the dynamic range of the weights necessary). All models are optimized with a fixed number of Stochastic Gradient Decent (SGD) iterations. Learning rate scheduler starts with a short warm-up phase followed by a cosine decay phase. For each optimization step a batch is drawn with replacement from the target dataset and then augmented using standard augmentations methods.
We apply several simple tweaks designed to take advantage of the identical structure of the teacher and student, under the assumption that a good low precision proxy exists in the vicinity of the reference model within the parameters space. Specifically, we use intermediate layer outputs to compute the smoothed-1 distance between the teacher and the student features, we named this loss Intermediate Quantization (IQ) loss. In addition we apply in-batch inputs mixing [29]. We found these generally lead to a more stable training convergence under extreme quantization and improve KD results for both syn-thetic and real samples.
Small scale experimental details. We wish to evaluate the applicability of synthetic samples as replacement to real data for model calibration and distillation. For this experiment we use ResNet44 and Wide-Resnet28-10 [28] on Cifar10, Cifar100 datasets respectively.
We start the process by following each of the proposed generation schemes to produce synthetic samples based on the provided models alone. For Inception scheme samples we use 5x5 standard normal Gaussian smoothing kernel i.e., N (0, 1)), and scale of 0.01. Next, each model is quantized and calibrated for 200 steps with batch size 256. Samples are drawn with replacement from a limited subset of the target dataset which contains 50 samples per class.
After initial calibration, we optimize the models for 16,000 iterations of SGD with batch size of 512 and 256 for ResNet44 and Wide-ResNet28-10 respectively. IQ loss is computed as the mean of the smoothed-l1 outputs of Conv-layers 2-5 from ResNet architecture, and multiplied with a scale of 0.001.
Small scale results. Results on post-calibration and KD fine-tuning are in table-1. We find that synthetic data optimized to reduce BN S is superior to other generation schemes, achieving comparable accuracy to real data even when fine-tune is required to recover from extreme degradation. Additionally, the Inception scheme is sensitive to the hyper-parameters choice (see appendix B). Finally, we observe a surprising outcome regarding the usefulness of Gaussian random samples under mild compression requirements for calibration and distillation. In our preliminary experiments, we noticed two fundamental components must be applied together to achieve our reported accuracy: (1) batch normalization freeze; (2) IQ loss (smooth-l1 in this case). We believe the full potential of this method is yet to be discovered and we encourage it as a future research direction. One simple path is truncating the model to disjoint intervals between BN layers then retrain each interval using KD over synthetic data generated by the applying Gaussian scheme on the statistics of the previous BN layer.
Large scale experimental details. To further demonstrate the applicability of our findings in a large-scale setting, we provide our results on ImageNet. ImageNet has been noted to be a challenging generative task even when full data access is granted, due to relatively large spatial config samples Training-Set  Table 1: Comparison of Cifar validation accuracy using low numerical precision models. Calibration using synthetic samples is on par with real data for more permissive quantization schemes, however, results reflect statistics becomes crucial factor at lower precision. Accuracy recovery via KD is comparable when using the real data and our BN S scheme.
size of the data and the number of classes [25]. Previous work [21] showed that 8-bit models can be calibrated without any data if BN layers exist. However, they did not investigate more challenging numerical precision levels, and thus did not need to fine-tune the model. We conduct our experiment with a pre-trained ResNet-18 from torchvision model-zoo [23]. First we generate 100k samples, then perform KD to fine-tune the quantized student, following similar settings to ones described in the previous section, with the exception of a longer regime of 44000 steps and a batch size of 256. Throughout our experiments the quantization scheme is fixed to a widely used method described in [15]. Methods that improve this scheme (such as bit allocation, bias correction [1], and equalization [21,19]) are orthogonal to this work and are expected to improve accuracy respectively.
Large scale results. Table 2 presents a promising result, both for calibration and for recovering model accuracy for ImageNet classification task. To the best of the authors' knowledge, this is the first time a compressed model was successfully fine-tuned without using any real data other than the trained model itself at this scale. We observe an accuracy gap (relative 1.5% degradation) between fine-tunning the model using standard Cross En-tropy (CE) objective to the KD variant when the full dataset is available, however, this gap is inverted when using significantly less data in favor of KD (detailed comparison in appendix, section C). We believe this can be attributed to a couple of factors. First, semi-supervised KD does not use the ground truth label information. Thus, final accuracy depends solely on the prediction quality of the teacher whereas label information can be used to penalise the student when repeating similar mistakes made by the teacher. Additionally, we speculate that a given bias in the reference model's prediction towards certain classes may degrade the student accuracy when training on raw teacher outputs.
Finally, we conclude that using synthetic data leads to either equivalent or slightly lower final accuracy compared to real data. We associate this gap with the synthetic data quality as well as failure to fully capture the dataset variety (i.e. mode-dropping [26]).    explanation to the success of BN S scheme for model compression over its counterparts. We will consider the differences between methods from the perspective of the optimization objectives for generating the synthetic data. In addition, we evaluate the impact of the internal statistics' divergence on KD potential for recovering lost accuracy. Monitoring Internal Statistics. As an initial step in our evaluation we perform a pair of experiments on Inception and BN S schemes. In each experiment we follow the process of one scheme and observe the other's objective behaviour. More specifically, we generate a batch of samples using Inception scheme while monitoring the internal statistics behaviour through J KL . For the alternative view, we generate a second batch of samples using BN S scheme, and observe the input smoothness loss. In both experiments we perform 1000 optimization steps on a batch of 128 samples. For each iteration we use in-batch augmentations with N=4, additionally a Gaussian kernel of size 3x3 and sigma=1 is used for the smoothing loss operation. Results are described in Figure 1.

Analysis of Data Generation Schemes
Under the Inception generation scheme, we find evidence for the divergence of the internal statistics as seen in figure 1a. Additionally, figure 1b indicates that minimizing J KL leads to lower smoothness loss. Interestingly, figure 1a the initial improvement in J KL during optimization of I hints to the existence of a connection in a reversed direction as well. However, smoothing loss alone proves to be insufficient in regularizing the internal statistics' divergence as the Inception loss greedily attempts to maximize the target class predictions and the measured J KL is increasing after several iterations.
BN-Stats and Model Accuracy. For the next part of our evaluation, we generated samples from the reference model using the BN S scheme. During the sample generation phase, snapshots of the training data were saved at different step intervals. The snapshots reflect different stages of the J KL loss curve. We then perform a series of KD experiments on the quantized reference model to observe the impact of dataset size and J KL loss value on the final model accuracy. Each experiment is repeated with five different seeds, the student model precision configuration is fixed to 4 bit activations and 2 bit weights except for the first and last layers which are using 4 bit.
Final results are presented in Figure 3 as follows: Figure 2b demonstrates that samples with lower statistic loss lead to better validation accuracy, while Figure 2a shows results are very close to real data with slight degradation under identical settings.

Data Correspondence: BNS Across Datasets
In this section we observe how the internal statistics of a model respond to different inputs, illustrating that similar input data should not lead to significantly different internal activation statistics. In table-3 we demonstrate this by measuring J KL of various datasets on a set of pre-trained ResNet44 [7] models. We also measure the J KL divergence in response to the original training data modified with Fast Gradient Sign Method (FGSM [6]). A small perturbation ratio = 0.1 is used to reduce model accuracy without considerably altering the perceived input. As expected from a similarity score, table-3 demonstrates that J KL maintains its proportion with respect to the original training dataset. Magnitude may change across models and datasets, we provide raw measurements in table-1 of the appendix (A). We further observe that in some cases, perturbed samples results in a significant increase in the internal statistics' divergence. This invites further exploration in the direction of possible applications of monitoring internal statistics to explain and detect adversarial attacks through tailored outlier sensitive measures.

Discussion and Future Work
In this work we considered the case of data-free KD for the purpose of model quantization. To achieve this goal, we suggested new methods for sample-generation and techniques to leverage them in a student-teacher training procedure. We have demonstrated the applicability of such samples for a true data free model compression in both small and large scale image classification tasks. Ultimately, this can open a path to improve data privacy by reducing the extent of real data exposure during production phase of deployment. To the best of our knowledge this is the first time a data-free distillation-based approach was applied to a model and achieved final accuracy results comparable with real samples under identical optimization conditions at scale. Our best performing method, which we referred to as BN-Stats, leveraged the common batch-norm layer to generate examples that mimic the per-channel mean an variance of each BN layer input. This was done by explicitly optimizing J KL divergence loss. Figure 1b shows that optimizing this loss, results with smooth input features, although we did not explicitly optimized for it, indicating a strong connection between BN-Stats loss and the local structure of the input. This invites further study to determine the viability of the method for reproduction of inputs other than images, where prior knowledge may be harder to apply directly. We also noticed that compared to BN S, inception scheme generates samples with a high model prediction confidence, yet cause the internal statistics' divergence to grow significantly (figure 1a). The success of BN S lead us to consider it as a possible measure for evaluating correspondence between a model, trained on a specific dataset and other viable alternative datasets.
We consider two drawback of the proposed method. One is the computational cost of generating samples through back-propagation, which can impede the practical use with large scale models for continuous train-deploy scenarios. However, we note that as long as the new training data does not significantly change, the internal statistics behaviour of the generated samples can potentially be shared -to avoid reproducing an entirely new synthetic dataset at each deployment cycle. We leave the exploration of cross model and cross dataset applications for future work. Second, we find the BN S method produces datasets which are unbalanced in terms of the mean output distribution of the reference model, due to the lack of explicit conditioning on the model output. However our experiments with BN S + I did not show a dramatic improvement despite their added balance control and the ad-

B Calibration and hyper-parameters sensitivity
Here we present evidence of hyper-parameter sensitivity for Inception related generation schemes. Specifically, we demonstrate how changing the variance parameter of the Gaussian smoothing kernel, used as the image prior to regularize the optimization process, impacts the usability of the generated data. In

C Additional results for ImageNet dataset
There exists an accuracy gap between the fine-tuned model and the distilled variants even when the full data-set is available as seen in table-6. We believe this can be attributed to using KD without an additional ground truth loss term which is common in unsupervised setting. Thus, final accuracy depends solely on the prediction quality of the teacher. Whereas additional label information can be used to penalise the student when repeating similar mistakes made by the teacher and contribute to an improved final accuracy. Additionally, we speculate that a given bias in the reference model's prediction towards certain classes (see figure-5) may degrade the student accuracy when training on raw teacher outputs.

D Generating Cifar10 and Imagenet samples
In figure-3, we provide a several synthetic samples per class generated from a Cifar10 trained ResNet44 model, using Inception scheme (I) and samples generated using the BN-stats + Inception scheme (BN S + I). The generation process follows the settings detailed in Generating data samples section. Samples seem to share similar visual features between generation schemes. while BN S + I samples are smoother and appear clearer. Additionally, a subset of samples generated using ResNet-18 model trained over ImageNet dataset is presented in figure-4. Visual inspection of those generated samples indicated a considerable improvement in detail and diversity over naive inception generation scheme. Furthermore, some instances seem to preserve the represented class physical structure better than others. Despite the lack of consensus regarding objective image quality evaluation methods, we find it is intriguing to further explore means to improve reproduced class visual quality and investigate its connection to the model's prediction quality. We believe BN S +I method may serve as a powerful tool for DNNs interpretability, providing insight into the featured learned by DNNs.

E Perceived dataset bias
Given a reference classification model, we loosely consider the per-class mean prediction of the model as unbiased if it is close to uniform when presented with a balanced set of examples, since no particular class should be favoured over other classes by an unbiased model. To provide a qualitative evaluation, we record the softmax output of a pretrained ResNet-18 model from torchvision, over the entire ImageNet validation set. Then, compute the mean prediction of the "soft" outputs (i.e., 1 N N i=1 output i ) where output i ∈ R #classes is the model softmax output for sample i, as well as the "hard" mean prediction (i.e., 1 N N i=1 e argmax{outputi} where e is the standard vector). In figures-5a,5b,5c, we plot the mean soft and hard predictions of the model for a given dataset, while classes are sorted according to the mean soft prediction. Figure-5a reveals that although the validation dataset is balanced (ignoring possible annotation errors and class similarity), the model produces a somewhat biased prediction. We hypothesize this bias may impede KD performance as discussed in Large scale experiment section, even when the entire dataset is available.
Additionally, in figures-5b,5c we provide the mean prediction plots for BN S and BN S + I datasets appropriately, each containing 100K synthetic samples. Figure-5b shows a clear preference towards certain classes, which does not necessarily align with the model's prediction bias over the validation set ( figure-5a). While figure-5c shows that BN S + I dataset appears balanced in terms of the mean prediction, which is not a surprising result. However, our KD experiments did not show any significant benefit to using BN S + I dataset compared to BN S dataset in terms of the final validation accuracy, despite the perceived class imbalance.

F Impact of under-represented classes
To further investigate the impact of BN S dataset bias on fined-tuned (quantized) model's accuracy with respect to poorly represented classes, we consider the mean relative accuracy degradation over N least frequent classes (mean tail degradation). First, we measure the per-class validation dataset accuracy for each of the fine-tuned models (i.e., ResNet18 fine-tuned with BN S or BN S + I dataset, see Large scale experiment section). Then, we compute the relative degradation with respect to the float model per-class validation accuracy. Classes are sorted in ascending order according to the float model mean prediction over BN S dataset (figure-5b). Essentially, least frequently predicted classes appear first in the ordered form, as perceived by the reference model. Finally, in figure-5d we show the mean degradation with respect to an increasing number of classes, while only considering classes where accuracy degradation is observed (i.e., ignore classes which present an improvement in accuracy). Figure-5d presents evidence for an improved mean tail degradation, which can serve as motivation for using BN S + I over BN S dataset, since the worst case accuracy is improved despite the overall accuracy being slightly worse.