{'fraction_non_alphanumeric': 0.02856185048608783, 'fraction_numerical': 0.010124036205162588, 'mean_word_length': 4.8586017282011, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Why Knowledge Distillation Amplifies Gender Bias and How to Mitigate from the Perspective of DistilBERT

Knowledge distillation is widely used to transfer the language understanding of a large model to a smaller model.However, after knowledge distillation, it was found that the smaller model is more biased by gender compared to the source large model.This paper studies what causes gender bias to increase after the knowledge distillation process.Moreover, we suggest applying a variant of the mixup on knowledge distillation, which is used to increase generalizability during the distillation process, not for augmentation.By doing so, we can significantly reduce the gender bias amplification after knowledge distillation.We also conduct an experiment on the GLUE benchmark to demonstrate that even if the mixup is applied, it does not have a significant adverse effect on the model’s performance.


Introduction
Knowledge distillation (Hinton et al., 2015) is one way to use the knowledge of a large language model under the limited resources by transferring the knowledge of a larger model to a smaller model. Under the supervision of the teacher model, the small model is trained to produce the same result as that of the teacher model. By doing so, small models can leverage the knowledge of larger models (Sanh et al., 2019).
To maintain the performance of the model trained by knowledge distillation, the distilled model focuses more on the majority appearing in the data (Hooker et al., 2020). Recent studies have described that pre-trained language model also results in a more biased representation when distillation proceeds (Silva et al., 2021). However, only the issue is reported, and what part of knowledge distillation causes an increase in bias is not explored, and no solution is provided. This paper studies which part of knowledge distillation causes the increase of social bias and how to alleviate the problem in terms of Dis-tilBERT (Sanh et al., 2019). We first examine what part that contributes to knowledge distillation brings social bias amplification. There is no difference between the distilled and original models except for size and training loss. Thus, we check from two perspectives: (1) the capacity of the model being distilled and (2) the loss used in knowledge distillation. Then we suggest leveraging mixup (Zhang et al., 2018) on the knowledge distillation loss to mitigate this amplification by giving generalizability during the training.
We conduct the experiments from two measurements: social bias with the Sentence Embedding Test (SEAT) (May et al., 2019) and downstream task performance with the GLUE Benchmark . We report that the factors that increase the social bias are the student model's limited capacity and the cross-entropy loss term between the logit distribution of the student model and that of the teacher model. We also demonstrate that applying the mixup to knowledge distillation can reduce this increase without significant effect on the downstream task performance.
Our contributions can be summarized as follows: • We reveal the capacity of the model and crossentropy loss in knowledge distillation have a negative effect on social bias.
• We suggest mixup as a mitigation technique if it is applied during the knowledge distillation proceeds.

Background
Knowledge distillation is trained so that a student model outputs the same output as a teacher model's for one input. It makes the student model have the problem-solving ability of the large model, even though the student model has a smaller structure.
DistilBERT, the model this study is mainly about, is trained with three loss terms. First, cross-entropy loss (L ce ) forces the logit distribution between the student model and the teacher model to be similar. Next, the student model learns language understanding itself with masked language modeling loss (L mlm ). Lastly, cosine loss between two model's output (L cos ) makes the direction of output embeddings between the student model and the teacher model closer (Sanh et al., 2019). In total, the loss term of DistilBERT is as follows:

Bias Statement
In this paper, we investigate stereotypical associations between male and female gender and attribute pairs, particularly from the perspective of sentence embeddings in knowledge distillation language models. For the attribute pairs, we consider Careers and Family, Math and Arts, and Science and Arts. If there exists a correlation between a certain gender and an attribute, the language model intrinsically and perpetually causes representational harm (Blodgett et al., 2020) through improper preconceptions. Additionally, when the language model is trained for other downstream tasks, such as occupation prediction (De-Arteaga et al., 2019;McGuire et al., 2021), it may lead to an additional risk of genderstereotyped biases.
Since knowledge distillation (KD) has become a prevalent technique to efficiently train smaller models, it is vital to figure out to what extent the gender biases are amplified after knowledge distillations and which loss terms exacerbate the biases during the training. Our work firstly conducts the in-depth analysis and then proposes mitigation methods for the gender bias amplification during the KD process.
We measure the streotypical associations with the Sentence Embedding Association Test (SEAT) (May et al., 2019) 1 . The SEAT uses semantically bleached sentence templates such as "This is a [attribute-word]" or "Here is [gender-word]". Then the associations between a gender and an attribute are calculated by cosine similarities of sentence encoded embeddings. We leave the detailed equations to calculate the SEAT scores in Appendix B.
There are several tests in SEAT. This study focuses on C6, C7, and C8 categories related to 1 https://github.com/W4ngatang/sent-bias/ gender bias. C6 tests similarity between embedding of Male/Female Names, and Career/Family attribute words. C7 and C8 measure the similarity between embeddings of male and female pronouns and embeddings of Math/Arts related words and Math/Science related words, respectively.

Gender Bias Amplification after KD
In this section, we conduct in-depth analyses about what brings gender bias amplification after knowledge distillation from the perspective of (1) the student model's capacity and (2) the loss used in the knowledge distillation process.

Experimental Setup
We use 30% of the corpus constructed by two datasets, the Wikipedia dataset and Bookcorpus (Zhu et al., 2015) dataset that were used to create DistillBERT 2 . The distillation is trained for three epochs using four V100 GPUs. All other settings remain the same following the way Distil-BERT is trained. We list the settings in Appendix D.

Does the capacity of the student model matter?
To figure out whether and to what extent the student model's parameter capacity affects the gender biases, we varied the number of layers of the student model (DistilBERT). Note that BERT and DistilBERT have the same architecture parameters except the number of layers. Figure 1 shows SEAT Loss Term L mlm + L cos + L ce L mlm + L ce L mlm + L cos C6 that the average SEAT scores are increasing as the number of layers is decreasing. Quantitatively, the number of layers has a strong negative correlation with the SEAT score (Pearson r = −0.82), which means that the smaller the capacity, the more severe the gender bias. This result also aligns with the previous study that reveals the models with limited capacity tend to exploit the biases in the dataset (Sanh et al., 2021).

Does the knowledge distillation process matter itself?
To ascertain how each loss term contributes to the increase in SEAT scores in the knowledge distillation process, we conducted an ablation study against each loss term. As shown in Table 1, the model trained without the distillation loss L ce results in the lowest average SEAT score (0.670) among the three loss functions. However, this model shows the lowest performance (75.2%) in the GLUE benchmark, whereas the model trained with all loss terms results the best with 76.7%. This implies that the transfer of the teacher's knowledge is helpful for general language understanding tasks while exacerbating gender bias simultaneously. Consequently, it can be concluded that the current knowledge distillation technique itself is also a factor in increasing gender biases.

Proposed method
This section describes how to improve the distillation process to make gender bias not amplified even after knowledge distillation. We found two causes (capacity, loss term) in the previous section. Among them, we decide to modify the loss term because this study is targeting the fixed size model, DistillBERT.
According to the ablation study in Section 4.3, we ascertain distillation loss (L ce ) hurts gender bias scores in a huge portion. Our intuition to alleviate this amplification is to give supervision as fair as possible during the knowledge distillation is proceeded. One way is to reduce the SEAT score of the teacher model first and give its supervision to the student model. However, most of the existing methods (Liang et al., 2020b;Cheng et al., 2021) for the teacher are designed to work only on the special token ([CLS]). It is not suitable for knowledge distillation that is trained with logits and embeddings on a token-by-token basis.
In this paper, we use mixup (Zhang et al., 2018) on knowledge distillation to increase genderrelated generalization ability by using mixup. Specifically, when a gender-related word appears, we use the values generalized by a mixup in the knowledge distillation process. First, we employ the pre-defined gender word pair (D) set (w male : w female ) from the previous work (Bolukbasi et al., 2016) 3 . We next make the teacher's output logit (y) and student's input embedding (x) same or similar between two corresponding gendered terms with λ drawn from Beta(α, α) when words in D appear: . We train DistilBERT with the mixup applied instances (x,ȳ) for words in D and with the original instances (x, y) for the rest of words. Notice that we do not use mixup as a data augmentation technique but rather employ its idea in the knowledge distillation.
We view the mixup as being worked as a regularizer rather than as a learning objective when knowledge distillation takes place (Chuang and Mroueh, 2021;Liang et al., 2020a). Because the student model learns masked language modeling itself, the generalized gender information by the mixup will act as a regularizer not to be trapped in the information commonly appearing in the pretraining corpus.

Experimental setup
Dataset We only use the same dataset in knowledge distillation used in Section 4. Also, we lever- 3 We list the pairs in Appendix C   age GLUE Benchmark to assess model performance.
Baseline We set a baseline as the distilled model from a teacher model that was trained with a debiasing method (Kaneko and Bollegala, 2021).

Experimental Results
In Table 2, we report the scores for each SEAT test and the average. It shows that mixup (Zhang et al., 2018) applied in the distillation process outperforms in terms of the average SEAT score. Compared to the baseline, distilled model under the supervision of the debiased teacher, mixup scores lower in four out of six tests (C6b, C7, C8, C8b). Table 2 also shows the results according to the part where the mixup is applied. We experimented with applying mixup to many different levels of representations in the distillation process: logits, teacher's output embeddings, and student's input embeddings. The proposed method that applies the mixup to inputs (input embeddings) and labels (logits) showed the best results.
We also measure SEAT after applying the teacher's output embeddings. It is because, although not included in the original distillation, the cosine loss for embedding is included in the learning process of DistilBERT. However, Table 2 reports that the mixup on output embeddings increases the SEAT score in most tests and is even higher than the original distillation process.
We also checked the performance on downstream tasks when mixup is applied in knowledge distillation. Table 3 summarizes the results on GLUE benchmark. Compared to the model using the original distillation, the average performance remains the same.

Conclusion
In this paper, we study what causes gender bias amplification in the knowledge distillation process and how to alleviate the amplification by applying mixup in the knowledge distillation process. We confirmed that both the cross-entropy loss between the logits and the model capacity affects the increase of gender bias. Since this study focused on the DistilBERT, we alleviated the problem by modifying the knowledge distillation loss. We reported that the SEAT score decreased when the mixup was applied to the student's input embedding and the teacher's output logit in the distillation method when gender-related words appeared. We also showed that this method does not have a significant adverse effect on downstream tasks.
There are limitations in this study. First, we used sub-samples of the pre-training corpus. Although we checked that there was no significant differences when trained with a fraction of data in terms of the SEAT score and the GLUE score, the experimental results for the entire data should be explored. Second, we do not yet know why the SEAT score increases when the mixup is applied to the output embedding. The embeddings between the two genders are expected to be close, but we do not yet figure out why the scores are reversed contrary to expectations. We leave these as our future work. 269 and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.

A Related Work
There were several attempts to apply mixup in knowledge distillation. Du et al. (2021) uses a fair representation created by the medium of the embeddings of two sensitive attributes (the neutralization) in distillation. Students are trained with the neutralized embeddings created in this way so that the student's input is dependent on the teacher's output. MixKD (Liang et al., 2020a) applies mixup during knowledge distillation to get better performance on the GLUE benchmark. Notably, MixKD takes the method of training the teacher model as well as the student model when distillation proceeds. Our suggestion guarantees independence between student and teacher model inputs in this work, as DistilBERT is trained. Moreover, we train a taskagnostic model by applying a mixup to distillation.

B Sentence Embedding Association Test (SEAT)
Let X and Y be target embeddings, the embedding of sentence template with gender word in our case, and A and B as attribute words. The SEAT basically measures similarity difference between attribute words and target word w. So the similarity difference on word w is