{'fraction_non_alphanumeric': 0.03493050475493782, 'fraction_numerical': 0.012948061448427212, 'mean_word_length': 4.884847180370211, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study

Mixup is a popular data augmentation technique based on creating new samples by linear interpolation between two given data samples, to improve both the generalization and robustness of the trained model. Knowledge distillation (KD), on the other hand, is widely used for model compression and transfer learning, which involves using a larger network's implicit knowledge to guide the learning of a smaller network. At first glance, these two techniques seem very different, however, we found that"smoothness"is the connecting link between the two and is also a crucial attribute in understanding KD's interplay with mixup. Although many mixup variants and distillation methods have been proposed, much remains to be understood regarding the role of a mixup in knowledge distillation. In this paper, we present a detailed empirical study on various important dimensions of compatibility between mixup and knowledge distillation. We also scrutinize the behavior of the networks trained with a mixup in the light of knowledge distillation through extensive analysis, visualizations, and comprehensive experiments on image classification. Finally, based on our findings, we suggest improved strategies to guide the student network to enhance its effectiveness. Additionally, the findings of this study provide insightful suggestions to researchers and practitioners that commonly use techniques from KD. Our code is available at https://github.com/hchoi71/MIX-KD.


Introduction
Deep neural networks have achieved impressive performance on a wide range of tasks including language translation [30,26], image classification [34,6], and speech recognition [2,19]. To further improve the model's efficiency and performance, a large number of training tech-niques have been proposed such as mixup-augmentation [38,35] and knowledge distillation [8]. In specific, mixup [38] is a commonly used data augmentation technique based on using convex combinations of samples, and their labels. This technique was introduced to improve generalization, as well as increase the robustness against adversarial examples. Recently, there has been an increasing interest in reducing the model size while preserving comparable performance, which narrows the gap between large networks and small networks. KD is one of the promising methods for this demand [8]. The goal of KD is to exploit the ability to learn concise knowledge representation (logit or feature) from a larger model and then embed such knowledge into a smaller model. For example, in image classification, deep neural networks produce class probability by using softmax function that converts logit f i into a probability p i by comparing f i with other logits as follows: p i = exp(fi/T ) j exp(fj /T ) . In the conventional knowledge distillation [8], the temperature T is utilized to generate a softer distribution of pseudoprobabilities among the output classes, where a higher temperature increases the entropy of the output, thus, providing more information to learn for the student model.
Motivation: On the surface, mixup and KD are very different, however, we found that "smoothness" turns out to be the connecting link between the two and a very important attribute to understanding KD's interplay with mixup. Intuitively, the KD involves a student mimicking smoothed probability distribution of the teacher, whereas mixup artificially introduces smoothness (the labels are not strictly one-hot) in a model via linear combinations of the labels. Although mixup-augmentation and KD are common techniques in training networks in various applications, the interplay between the two has not been well explored. In this paper, we investigate in detail the impact of mixupaugmentation on knowledge distillation.
To develop insight into several interesting behaviors of the network, we provide various visualizations in feature and logit levels. For example, to analyze how mixup enforces a feature representation in-between each class, we pick 5 semantically similar classes from the CIFAR100 dataset (baby, boy, girl, man, and woman). Then, we project the features extracted from the penultimate layer into 2-D using tsne [29] as seen in Figure 1. The first row of the figure represents the feature representation for two networks trained from scratch with mixup (Mix-S:ResNet110 or Mix-S:ResNet56) and without mixup (S:ResNet110 or S:ResNet56). Generally, we observe that the higher capacity model (S:ResNet110) encourages the deep networks to learn tight projections in each class while the lower capacity model (S:ResNet56) learns more diffuse projections. This is also verified by clustering metrics, such as the V-score [21], where a higher value implies better clustering. Interestingly, the mixup-trained models disperse features in similar classes, compared to the one trained without mixup, even though a mixup-trained model shows better test accuracy due to an increase in generalization on unseen data. Meanwhile, we found similar observations on the feature representation from distilled models when a higher temperature is used (i.e., distilling more softer logit) as seen in the second row of Figure 1. From the table in Figure 1, high temperature T is a default choice to improve the performance. In this way, the teacher transfers more information to the student even though higher temperatures superficially promote feature dispersion in similar classes. However, transferring a high quality of supervision to a student network is also crucial as it can guide the student to learn discriminative representations from the superior teacher network. Thus, we seek a way to increase supervisory signals from teacher to student, without compromising on performance. This is where mixup presents a way forward. In short, the crux of our insight is that if a teacher is trained with smoothed data from mixup, then further 'smoothing' at high temperatures during distillation can be avoided. This ensures stronger supervisory signals while enjoying the benefits of dataset augmentation via mixup. We summarize our contributions as follows: 1. We provide new insights into devising improved strategies for learning a student model, by a deeper understanding of the behavior of features/logits and networks trained with mixup during KD.
2. To reduce the criticality of choosing a proper 'temperature', we develop a simple rescaling scheme that brings different statistical characteristics in logits between teacher and student to a similar range while preserving the relative information between classes, thereby the temperature T is no longer used.
3. We identified that strongly interpolated mixup pairs impose extra smoothness on the logits, thus we can generate only a few mixup pairs in a batch, called partial mixup (PMU), and yet achieve comparable and even better performance with this variant in KD.

Background
We first introduce the background of mixup and KD through a simple mathematical description. Given the training data D = {(x 1 , y 1 ), ..., (x n , y n )}, the goal of the classification task is to learn classifier f : X → R k by mapping input x ∈ X ⊆ R d to label y ∈ Y = {1, 2, ..., K}. Let L(f (x), y) be the loss function that measures how poorly the classifier f (x) predicts the label y. Mixup Augmentation [38] In mixup augmentation, two samples are mixed together by linear interpolation as follows:x ij (λ) = λx i + (1 − λ)x j , andỹ ij (λ) = λy i + (1 − λ)y j , where λ ∈ [0, 1] follows the distribution P λ where λ ∼ Beta(α, α). Then, the mixup loss function can be described as where L represents the cross-entropy loss function in this study. Specifically, a hyper-parameter λ in Equation 1 is used to specify the extent of mixing. In other words, the control parameter α in a beta distribution commands the strength of interpolation between feature-target pairs, i.e., the high α generating strongly interpolated samples. A fair amount of variants of mixup have been proposed [31,35,11]. The general strategy of these mixing-based methods is intrinsically similar in that they extend the training distribution by blending the images and mixing their labels proportionally. Thus, in this study, we only pay attention to the conventional mixup to investigate the interplay between mixup and knowledge distillation [38]. Knowledge Distillation [8] In knowledge distillation, given a pre-trained teacher model f T on the dataset in advance, the student model f S is trained over the same set of data by utilizing concise knowledge generated by f T . In specific, once the teacher network is trained, its parameter is frozen during the training in KD, and then, the student network is trained by minimizing the similarity between its output and the soft labels generated by the teacher network. To this end, we minimize the divergence between logits of the student and the teacher as follows: where S indicates the softmax function, KL measures the Kullback-Leiber divergence, and output logits of the model smoothen by temperature, T .

Data Augmentation in Knowledge Distillation
Recently, several works have utilized data augmentation and achieved promising results in that augmented samples enable the networks to learn relaxed knowledge from different views in distillation frameworks [33,32,12]. Across the broadly available methods, they often do not provide insight into the inner working of the models. Meanwhile, recent works [15,23] have studied KD's compatible/incompatible views with label smoothing [25] to provide such insight into deep models via empirical analysis. Instead, we are interested in the underlying mechanism of augmentation in the light Figure 2. The student (fixed RN56) test accuracy on CIFAR100 under different teacher networks with T = 4. Here, T and S denote the standard teacher and the standard student. Mix-T and Mix-S are mixup-trained teachers and mixup-trained students, respectively. Mixup-trained teachers guide lower-performance students than the one with standard teacher models, and lowercapacity teacher models generally distill inferior students.
of the KD process. Our study spans double aspects that attempt to present both compatible and incompatible views through comprehensive empirical analysis. Further, based on the observations, we suggest a better learning strategy to enhance the network's performance.

Key findings from Mixup and KD interplay
In this section, we discuss our main findings. We first begin by referring to Figure 2. Here, we describe four possible scenarios where mixup-augmentation could be involved in KD as follows; standard teacher and standard student (T&S), mixup-trained teacher and standard student (Mix-T&S), mixup-trained teacher and mixup-trained student (Mix-T&Mix-S), and standard teacher and mixup-trained student (T&Mix-S) under the same temperature being T = 4. We fix the student model as ResNet56 (RN56) and evaluate it by varying the teacher models from ResNet20 (RN20) to ResNet110 (RN110).
As seen in Figure 2, we can make two observations: First, the student performance with the help of a mixuptrained teacher (Mix-T&S, Mix-T&Mix-S) always shows less effectiveness than with the help of a standard teacher (T&S, T&Mix-S) under the same settings of temperature even if the mixup-trained teacher itself shows better test accuracy than the teacher trained without mixup. Second, generally, higher capacity teacher models distill better students, but with a lower capacity teacher (T:RN44 & S:RN56), the student performance still shows improvement in accuracy, compared to the vanilla student model (green dashed line, S (No KD)). Based on these observations, this paper aims to investigate the following questions: 1) Why does a mixup-trained teacher model impair the student's effectiveness in KD? We answer this question in section 3. Then, 2) what can we do to improve the effectiveness of knowledge distillation when mixup-augmentation is applied? We address this question in section 4.

Observation 1) Mixup vs Non-Mixup.
To investigate the effect of the network trained with mixup-augmentation, we selected a few classes and divided them into two groups: (1) semantically different classes (Beaver, Apple, Aquarium Fish, Rocket, and Turtle) and (2) semantically similar classes (Baby, Boy, Girl, Man, and Woman), all from CI-FAR100. For example, in Figure 3, the left figure illustrates the feature representation of the penultimate layer on train sets. If we look at the number 1 in the red circle, the projections of a mixup-trained model from similar classes are more dispersed while the ones of different classes are still well-preserved in their structure. This information loss can also be measured by clustering metric, V-score on the right histogram of the figure, resulting in a drastic drop in V-score for similar classes.
Observation 2) Distilling from low-accuracy teacher models. If the student network is settled, retaining the fine quality of supervision (i.e., the performance of a teacher network) is crucial in training better students. As followed in the number 2 in the red circle, the projections of the student with the lower-accuracy teacher are notably dispersed, significant drop in V-score, eventually impairing the student's performance. This implies that the better student is distilled with the help of discernible features given by the high-capacity teacher.
Observation 3) Distilling from a mixup-trained teacher network. Observation 1) showed that the mixup-trained model scatters the features in similar classes. We now look at the case where a mixup-trained teacher conveys the knowledge to the student. In KD, as the student and teacher models are trained on the same train set, we argue that a student trained with supervision by a mixup-trained teacher cannot take advantage of learning superior knowledge due to feature scattering. As shown by number 3 in this figure, feature representations of similar classes in the student network eventually become more spread out, and V-scores also drop on both train and test sets. We provide the visualizations of train and test sets in the supplementary material.  Figure 4, where the entropy is popularly used to measure the smoothness of the distribution. The higher entropy value is the smoother distribution.
Here, we observe two intriguing phenomena; Observation 4) A mixup-trained model produces softer output logits, illustrated by the short red bars in both train and test set. From this observation, we surmise that a mixupaugmentation involved in training the student in KD contributes extra smoothness to the logits. Further, the student learned from a standard teacher outperforms the mixuptrained teacher in terms of accuracy (74.98% vs 73.92% and 71.78% vs 70.60%) under the same settings of the temperature T = 4. Conclusively, when distilling from a mixuptrained teacher, the use of high temperature adversely impacts the accuracy of the student. Observation 5) When the inferior knowledge produced by the low-accuracy teacher (T:RN20) transfers to the student, the confidence level on the prediction significantly falls as seen in the gray bars on both sets, leading to a remarkable decrease in test accuracy on both cases (74.98% → 71.78%, 73.92% → 70.60%). It implies that transferring decent quality of supervision to the student is crucial to successful knowledge distillation. Observation 6) Mix-T&S at low temperature. From observation 5), while increased T is believed to be helpful to produce better representation for KD, we remark that in the presence of a mixup-trained teacher, an increased T can be an adverse effect on the performance of KD because of the feature dispersion/exorbitant smoothness in the logit. At this point, one might give rise to the following question.
What if we lower the temperature to lessen smoothness in the logit?  Figure 5. Visualization with the heatmap representation. When distilling from a mixup-trained teacher (Mix-T) at a higher temperature, it transfers inferior knowledge to the student. With lower temperature T = 1, it produces better students, covering the relative high-accuracy region in the heatmap.
To investigate how much temperature impacts the test performance in KD, we visualize the test accuracy as a heatmap in Figure 5. In this figure, we explore all combinations of a mixup-trained teacher and student. The left panel of this figure shows test accuracy under the setting T = 4 and the right panel is for T = 1. We note that when a lower temperature, that makes the logit less smooth, is applied, the better the quality of knowledge that is transferred to the student. This result demonstrates that knowledge from a mixup-trained teacher at high temperatures renders the student model less effective.

Effective learning strategies for Mix-KD
Based on our findings, we observe that smoothness in the logit is a critical attribute in distillation, as seen in Figure 5. Since it is difficult to measure suitable smoothness from the perspective of the student, many distillation methods have relied heavily on a naive brute-force search to find the proper temperature. To reduce the criticality of choosing the proper temperature and alleviating excessive smoothness by a strong mixup, we introduce an effective learning strategy to improve the performance of KD. Partial mixup. The control parameter in the beta distribution plays a key role in controlling the strength of interpretation, which also affects the degree of softened output. As α → 1, it provides softer output logits. However, there is a trade-off between avoiding excessive smoothness and improvement in robustness against adversarial attacks in knowledge distillation. To alleviate this issue, we suggest generating only small amounts of mixup pairs used in training, called partial mixup (PMU). For example, PMU=10% refers to only 10% mixup pairs used in a batch and the rests keep untouched. To further understand the behavior of how partial mixup affects output probability, we provide toy examples using 2 classes in the supplementary material. Rescaled logits. Here, we suggest using standard deviation as temperature such that the logit produced by the output layer of the network is rescaled by dividing it by the standard deviation of that logit, thereby the temperature T is no longer hyper-parameter. This way can bring two different statistical characteristics between the teacher and the student logits to similar ranges while it does not hurt the relative structure in-between classes. We underline the importance of rescaling logit since the random mixing portion λ from a beta distribution governs the smoothness degree, yielding irregular smooth output at each iteration. Therefore, we replace the output logit with rescaled one, and the loss function becomes as follows: )), (3) wheref = f /σ(f ), S indicates the softmax function, KL is the Kullback-Leibler divergence, and σ(·) is standard deviation of input logit. Then, the final training objective for the student in KD is as follows: (4) where γ kd and α kd are balancing parameters. Note, the hyper-parameter of the partial amount in PMU is not denoted in this equation. Choice of α and PMU. Further, we investigate how each hyper-parameter (partial amount in PMU and the control parameter α) affects the student performance in KD under the following combination, T:RN110 and Mix-S:RN20 on the CIFAR100 dataset. We evaluate the student performance by varying the degree of partial amount from 10% to 80% with different control parameters in the range of [0.2, 0.4, 0.7, 1.0]. Figure 6 indicates that PMU with high α generally leads to better students in this setting. The lowest performance is observed in FMU (fully mixup pairs used) with α = 1.0. We note that PMU does not necessarily aim to outperform FMU, but it plays a regulatory role to adjust smoothness finely, thus helping us to understand the effect of smoothness in KD. We present performance analysis in section 5.1 and also further study how partial mixup responds to adversarial attacks in section 5.3.

Experiments
In this section, we present the experimental results to validate our conjectures and findings. In the previous section, we noted that (1) a mixup-trained teacher generates an inferior student compared to the teacher trained without mixup under the same temperature (2) but lowering temperature can recover the test accuracy. In KD, mixup augmentation is only useful for the teacher if a mixup-trained teacher provides the student with further benefits such as better knowledge and additional power of robustness. Through our analysis, we see that the expected effect of the use of mixup-trained teachers is not satisfactory. Therefore, we use the teacher trained without mixup for our experiment, i.e., T&Mix-S.
All the models are trained for 240 epochs with a learning rate of 0.05 decayed by 0.1 every 30 epochs after 150 epochs. The balancing parameters γ kd and α kd are 0.1 and 0.9 for all settings, respectively. In Table 1, we report our results of four different settings as follows, distilled models without mixup (No Mixup), distilled models with PMU (10% and 50% when α = 1), and distilled models with full mixup (FMU, α = 1). As seen in Table 1, the students trained with PMU consistently outperform the ones trained without mixup. Surprisingly, in some cases (e.g., T:W40-2 & S:W16-2 and T:W40-2 & S:SN1), our student trained with PMU performs better than the teacher. Experiments on ImageNet: Table 2 shows the top-1 accuracy on ImageNet [3]. In this experiment, we choose ResNet34 and ResNet18 [5] as the teacher network and student network, respectively. We train the model for 100 epochs and with initial learning rate is 0.1 decayed by 0.1 at 30, 60, and 80 epochs. The batch size is set as 256. For comparison with other distillation methods, the hyperparameters of other methods follow their respective papers. The balancing parameters γ kd and α kd are 0.1 and 0.9 respectively, and we report the partial mixup of 10% and 100% (FMU). We observe that the proposed method with full mixup boosts the top-1 and top-5 accuracy by 2.07% and 1.56% over the baseline and fully mixup shows better performance than 10% partial mixup. We will discuss lower performance cases in the following section. Performance analysis: Based on Table 1, in some of   Table 1. CIFAR-100 test accuracy (%) of student networks trained with several distillation methods. The results of the other distillation methods except * are quoted from [27]. The best and second-best results are highlighted in Bold and Red. * is performed with our implementation based on the author-provided code. the teacher-student combinations, we observe that adding more mixup pairs helps the student achieve higher accuracy than small amounts of pairs. Furthermore, for different architectural styles, the use of full mixup outperforms others in some cases. The presumption from this observation is that the networks from different architectures try to seek their solution paths, which means that the teacher and the student have unalike distributions in the logits, and thus imposing extra smoothness induced by strong augmentation might provide the student with additional information about how the different style teacher represents knowledge. Also, in the case of ImageNet having a large number of classes, the generated knowledge learned by two-mixing images and their labels might produce relatively less informative knowledge compared to CIFAR-100, so a strong smoothness by full mixup is favorable to distilling a better student model in this case. While we heuristically present that the performance is controlled by the amount of smoothness, how much exact smoothness should be imposed across datasets or networks remains an open question that can form the basis for further future work.

Robustness to adversarial examples
One undesirable consequence of models trained is their fragility to adversarial examples [4]. Adversarial examples are made by adding tiny (visually imperceptible) perturba- Table 3. Classification accuracy against the white-box attack with various perturbations of ϵ for every pixel. All methods are trained with full mixup (α = 0.2) and our distillation is trained with partial mixup only 10% (α = 1.0) on CIFAR100.   [4]. For I-FGSM, we use 10 iterations with an equal step size. The results for both attacks are summarized in Table 3. For distillation methods, the setting of teacher and student is T:RN110 & S:RN20, where they were trained on CIFAR100. We applied mixup augmentation to all methods we explored (see.  ). We trained our model using a partial mixup of only 10% with α = 1. Even with only 10% mixup pairs used, it shows impressively resistant to both attacks. We notice that distillation methods with the mixup, which utilize feature maps such as ICKD and CRD underperform those without mixup. We will show more comparative results of different amounts of mixup pairs in the next section.

Ablation Study
In this section, we conduct an ablation study for hyperparameters with a combination of T:WN40-2 & S:WN16-2 networks. As shown in Table 4, the distilled models with PMU and high α value generally yields better performance. Further, to investigate how much the degree of partial mixup and α react against both attacks, we also show test accuracy in Figure 7. As shown in the first and second columns of the histogram, when strong augmentation is involved in training such as full mixup (FMU with a high value of α) or the high number of mixup pairs (PMU 80%), it improves robustness. Interestingly, only 10% partial mixup (gray bar in the middle column) defends both attacks well. For the last column of the figure, we selected two distilled models from FMU (α = 0.2) and PMU (50%, α = 0.7) where they have similar test performance in Table 4. We observe that PMU (50%, α = 0.7) shows slightly higher robustness than FMU (α = 0.2).

Conclusions
In this work, we study the role of a mixup in knowledge distillation. We observed that a mixup-trained teacher network produces inferior supervision due to exorbitant smoothness imposed on the features and logits, especially at high temperatures during distillation. Thus, the students experience a reduction in their performance in KD. We sup- port our findings through a series of empirical analyses and large-scale experiments on the image classification task. Our findings provide insight into the inner workings of the distilled model trained with mixup augmentation. These insights allow us to develop an improved learning strategy using rescaled logits and partial mixups. As we mentioned earlier, various mix-based augmentations have shown their effectiveness for particular tasks. However, these augmentations may tend to create unreasonable training samples as it blends random images [11], which could distort reasonable relative structure among categories. As a result, this potentially yields an unfavorablesmooth effect on logits during distillation. Therefore, developing an augmentation method that automatically selects more reasonable samples ensuring the best-fitting smoothness would further boost the progress in the distillation field. We will further develop this technique in our future work.