{'fraction_non_alphanumeric': 0.03918050672808825, 'fraction_numerical': 0.012219662989453267, 'mean_word_length': 4.7317954419121735, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Balancing out Bias: Achieving Fairness Through Balanced Training

Group bias in natural language processing tasks manifests as disparities in system error rates across texts authorized by different demographic groups, typically disadvantaging minority groups. Dataset balancing has been shown to be effective at mitigating bias, however existing approaches do not directly account for correlations between author demographics and linguistic variables, limiting their effectiveness. To achieve Equal Opportunity fairness, such as equal job opportunity without regard to demographics, this paper introduces a simple, but highly effective, objective for countering bias using balanced training. We extend the method in the form of a gated model, which incorporates protected attributes as input, and show that it is effective at reducing bias in predictions through demographic input perturbation, outperforming all other bias mitigation techniques when combined with balanced training.


Introduction
Natural Language Processing (NLP) models have achieved extraordinary gains across a variety of tasks in recent years. However, naively-trained models often learn spurious correlations with other demographics and socio-economic factors (Hendricks et al., 2018;Lu et al., 2018;Bolukbasi et al., 2016;Park et al., 2018), leading to disparities across author demographics in contexts including coreference resolution, sentiment analysis, and hate speech detection (Badjatiya et al., 2019;Zhao et al., 2018;Li et al., 2018a;Díaz et al., 2018).
While balancing methods have been shown to be successful, they have not been tested extensively in NLP. In this paper, we focus on Equal Opportunity fairness (EO: Hardt et al. (2016)), which requires non-discrimination across demographics within the "advantaged" outcome labels, and adapt three balanced training approaches for debiasing. In addition, we propose a new objective for balanced training, which can be used for proxy optimization of EO fairness. We first provide a theoretical justification for our approach, and then conduct experiments on two benchmark datasets which show that our proposed objective is highly effective at achieving EO fairness while maintaining competitive accuracy.
Even when the training data is balanced, ignoring demographic-specific features can lead to bias (Wang et al., 2019;Lahoti et al., 2020), due to differences in language use across demographics (Hovy, 2015). There is thus a fine line to be walked in terms of optimizing for linguistic variables associated with different demographic groups (potentially boosting overall model accuracy), and ensuring model fairness.
Inspired by work in domain adaptation on learning domain-specific representations that generalize across domains (Bousmalis et al., 2016;Li et al., 2018b), we propose a gated model, which incorporates author demographics as an input to generate group-specific representations but also generalizes across demographic groups. We show that when combined with instance reweighting during training, this technique leads to substantial bias reductions over leading debiasing techniques, typically with higher predictive accuracy. We also introduce a second means of bias reduction through tailoring gating coefficients of the trained model, which allows for fine-tuning of the accuracy-fairness trade-off. Our experiments over two benchmark datasets for language debiasing show that our techniques are competitive with much more complex state-ofthe-art methods for debiasing in situations where the demographic attribute is not known at test time, and provide substantial gains over the state-of-theart when the protected attribute is observed.

Balanced Training
Despite their simplicity and versatility, balanced training approaches have received limited attention in prior work in NLP. In this section, we propose a novel objective for balanced training, which we show to be a proxy for the EO. We further review three balanced training approaches, discuss their objectives, and highlight their differences over our proposed method.

Problem Formulation
In this paper, we focus on bias mitigation for NLP classification tasks. Formally, we assume a dataset D = {(x i , y i , g i )} n i=1 where x i ∈ X is a d-dimensional input text representation vector, y i ∈ Y denotes the main task label (e.g., sentiment), and g i ∈ G represents the private attribute associated with x i (e.g., author gender).
A standard model M is trained to predict Y given X, while debiasing methods generally aim to learn a model M that is fair wrt G by considering X × G together. Let X be the task loss and n be the number of observed instances in the dataset D. The overall empirical risk is written as L = 1 n i X (y i ,ŷ i ), which can be rewritten as the aggregation of subsets: L = y g ny,g n L y,g , where n y,g is the number of instances with target label y and demographic attribute g, and L y,g is the empirical loss corresponding to the subset: L y,g = 1 ny,g i X (y i ,ŷ i )1(y i = y, g i = g). Furthermore, we use * to denote marginalization, for example, n * ,g = y n y,g . Let p be the target label distribution, andp be the empirical probability based on the training dataset.

Fairness Measurement
Equality of Opportunity (EO) is widely used in previous work (Hardt et al., 2016;Ravfogel et al., 2020;Han et al., 2021a), and measures the difference in true positive rate (TPR, aka Recall) across all groups, based on the notion that the positive outcome represents 'advantage', such as getting a job or a loan. Essentially, the difference (gap) in TPR reflects the degree to which different groups lack equal opportunity (with lower numbers indicating greater equity).

Towards Equal Opportunity
Without loss of generality, we illustrate with the binary case of y ∈ {T, F } and g ∈ {0, 1}. Recall that the equal opportunity metric is satisfied if a binary classification model has an equal positive prediction rate for the advantaged class. Assuming the advantaged class is y = T , the equal opportunity is measured by the TPR gap between protected groups, i.e., Recall g=0 − Recall g=1 . Our proposed objective function for equal opportunity is: Compared to the vanilla objective, the weights of instances with target label T are adjusted. Specifically, the reweighting term n T, * 2n T,g > 1 for the minority group, and < 1 for the majority group.
From CE to TPR Cross-entropy is an estimate of the TPR at the mini-batch level when considering a subset of instances with the same target label. Recall that the CE loss for binary classification of an instance is −[y i · log(p(y i )) + (1 − y i ) · log(1 −p(y i ))], wherep(y i ) is the predicted probability of y i being True. Taking y = T for a certain demographic group g as an example, Essentially, minimizing L T,g is proportionate to maximizing the TPR of demographic group g. That is, at the minibatch level, −L T,g is an estimator of log(p(ŷ = T |y = T, g = g)), which is the log-TPR of group g. Given this, our proposed objective minimizes the TPR gap by focusing on the log-TPR difference across demographic groups.
Beyond binary labels & demographic attributes Our proposed objective generalizes to higher ordering labels and demographic attributes trivially. The equal opportunity metric was originally designed for binary classification, under the Figure 1: Balanced training through downsampling w.r.t. different objectives over a toy dataset for illustrative purposes. N/P = main task, 0/1 = protected attribute. The groups at either end of each arrow are resampled to be the same size. EO = our proposed objective in Section 2.3. BD, CB, and JB refer to balanced demographics, conditional balance, and joint balance, resp. (Section 2.4).
assumption of a single advantaged class y = T .
To satisfy the multi-class target label case, we adjust the equal opportunity to consider the one-vsall setting, and measuring the TPR of each target class. Our proposed objective then becomes y g ny,g n ny, * |G|×ny,g L y,g .

Balanced Training Objectives
We now formally describe the objective functions of three established balanced training approaches, and discuss their applications. We provide a toy example in Figure 1 to illustrate the differences between these objectives. In Appendix E, we provide more details about the mapping from previous work to these objectives in our framework.
(2018) augment the dataset according to the demographic label distribution (making p(G) uniform) for bias mitigation in the context of coreference resolution. Although their gender-swapping approach is not directly applicable to our tasks, we adapt the general objective function as L G = 1 |G| y g ny,g n * ,g L y,g , where |G| is the number of distinct labels of G.
Since L G only encourages the model to equally focus on different demographic groups, it does not explicitly capture the correlation between G and Y , and as a result, does not achieve Equal Opportunity fairness.
Conditional Balance (CB) In a vision context, Wang et al. (2019) down-sample the majority de-mographic group within each class, so that on a per-class basis, it does not dominate the minority group (i.e. p(G|Y ) is uniform for all Y ), giving the objective function: L G|Y = 1 |G| y ny, * n g L y,g . This is the closest formulation to ours, as it also captures the conditional independence between G and Y . However, it captures both the TPR and TNR, while our method and EO fairness only focus on the TPR. In the multi-class target label case, our EO objective recovers the formulation of L G|Y .
Joint Balance (JB) Lahoti et al. (2020) employ instance reweighting for structural data classification such that demographics and classes are jointly balanced, leading to the objective: L G,Y = 1 |G|×|Y | y g L y,g . JB can be treated as a combination of the classic long-tail learning objective and the CB objective (p(G, Y ) = p(G|Y )p(Y )). On the one hand, JB is equivalent to CB when Y has already been balanced, which is the case for the dataset MOJI (Section 4.2), and CB is not a suitable objective for achieving EO fairness in this case. On the other hand, when Y is imbalanced, JB not only requires CB but also focuses more on long-tail main task classes, making it highly vulnerable to the size of minority groups.

Achieving the Objective
In this paper, we focus on two ways of achieving the target objective: (1) instance reweighting, which manipulates the weight of each instance during training; and (2) down-sampling, which preprocesses the dataset before training.
Taking the joint balance (JB) as an example, instance reweighting reweights each instance in inverse proportion to the frequency of the combination of its main label and demographic label, , where X is the task loss, andŷ i denotes the model prediction given input text x i .
The other approach, down-sampling, subsamples non-minority instances to derive a balanced training dataset, such thatp(g, y) = 1 |G|×|Y | , ∀g ∈ G, y ∈ Y . Specifically, let D y,g = {(x i , y i , g i )|y i = y, g i = g} n i=1 denote a subset for training. We sample without replacement to get a target subset D * y,g such that |D * y,g | = min{|D y ,g |, ∀y ∈ Y, g ∈ G}. The sampled subsets are merged to form the training set.

Demographic Factors Improve Fairness
Ignoring demographic-specific features can lead to bias even when the training data has been balanced (Wang et al., 2019;Lahoti et al., 2020). Instead, as suggested by Hovy and Yang (2021), addressing demographic factors is essential for NLP to get closer to the goal of human-like language understanding, and increase fairness. Our approach to dealing with this is, rather than removing demographic information, to use a gated model that uses demographic labels as input.
As can be seen in Figure 2, the gated model consists of (1+|G|) encoders: one shared encoder, and a dedicated encoder for each demographic group in G. 2 Formally, let E denote the shared encoder, E j denote the encoder for the j-th demographic group, C denote the classifier, and g i be a 1-hot input such that g i,j is 1 if the instance (x i , g i , y i ) belongs to the j-th group, and 0 otherwise. The prediction for an instance is: The two inputs are concatenated and input to the classifier C.
Intuitively, the shared encoder learns a general representation, while each group-specific encoder captures group-specific representations.
Our setting differs from other debiasing methods in that we assume the demographic attribute is available at training and prediction time, while techniques such as adversarial training (Li et al., 2018a) and INLP (Ravfogel et al., 2020) only require the attribute for training. This richer input allows for more accurate predictions, courtesy of the demographic-specific encoder, but limits applica-2 Strictly speaking, it is possible to achieve a similar effect with |G| encoders by merging one group with the shared encoder, and using post-hoc correction to separate out the general from the group-specific representation (Kang et al., 2020). bility at test time. For better applicability, we also relax this requirement by replacing demographic factors with a non-informative prior in Section 4.7.

Experimental Results
In this section we first introduce our experimental settings, and then report and discuss our results. In Appendix B, we provide detailed settings for reproducing our experiments.

Evaluation Metrics
Following Ravfogel et al. (2020), we use overall accuracy as the performance metric, and the separation criterion to measure fairness in the form of TPR GAP and TNR GAP: the true positive rate and true negative rate differences between demographic groups. For both GAP metrics, smaller is better, and a perfectly fair model will achieve 0. For multi-class classification tasks, we follow Ravfogel et al. (2020) in reporting the quadratic mean (RMS) of TPR GAP over all classes. In a binary classification setup, TPR and TNR are equivalent to the TPR of the positive and negative classes, respectively, so we employ the RMS TPR GAP in this case also.
Throughout this paper, we report accuracy and GAP results as mean values ± standard deviation over the test set, averaged across five independent runs with different random seeds.
In contrast to single-objective evaluation, evaluation of fairness approaches generally reports both fairness and performance at the same time. Typically, there is no single method that achieves both the best performance and fairness, making comparison between different fairness methods difficult. This problem has been widely studied in the literature on multi-objective learning (Marler and Arora, 2004). For ease of comparison between approaches, we adopt the compromise solution (Salukvadze, 1971) to fairness evaluation, and introduce 'distance to the optimum' (DTO). Specifically, the compromise solution aims to minimize the difference between the candidate point and a utopia point. In our case, the candidate points are ordered pairs (Accuracy, GAP), denoting the accuracy and fairness of debiasing methods, and the utopia point (optimum) represents the hypothetical system which achieves the highestachievable accuracy and fairness for the dataset., i.e., (max(Accuracy), min(GAP)). Following Vincent and Grantham (1981); Vincent (1983), DTO is calculated as the Euclidean distance between the optimum and the results for a given method. Lower is better for this statistic, with a minimum of 0. In Appendix C, we provide a more detailed explanation of DTO, including a step-by-step example calculation.
In addition, we are also interested in the efficiency of the different debiasing approaches and report each method's average training time. 3

Dataset
Following Ravfogel et al. (2020), we conduct experiments over two NLP classification tasks -sentiment analysis (MOJI) and biography classification (BIOS) -using the same dataset splits. In Appendix A, we provide analysis of the dataset distribution.
MOJI: This sentiment analysis dataset was collected by Blodgett et al. (2016), and contains tweets that are either African American English (AAE)like or Standard American English (SAE)-like. Each tweet is annotated with a binary 'race' label (based on language use: either AAE or SAE), and a binary sentiment score determined by (redacted) emoji contained in it.

BIOS:
The second task is biography classification (De-Arteaga et al., 2019), where biographies were scraped from the web, and annotated for binary gender and 28 classes of profession.

Models
We first implement a "STANDARD" model on each dataset, without explicit debiasing. On the MOJI dataset, we follow Ravfogel et al. (2020)  as their STANDARD model for the BIOS dataset, taking the 'CLS' token as the source of a fixed text representation, without further fine-tuning. However, we found that taking the average of all contextualized token embeddings led to an accuracy improvement of 1.4% and GAP fairness improvement of 2.4%. Given this, we use 768d 'AVG' representations extracted from the pretrained uncased BERT-base model.

Balanced Training Approaches
Since the MOJI dataset has been artificially balanced for main task and demographic labels, balanced training based on p(g) makes no difference, and moreover, the results for p(g|y) and p(g, y) will be identical. Given this, we focus on the BIOS dataset for comparing different balanced training objectives. 4 Table 1 shows the results of balanced training using the different objectives. Compared to the STANDARD model, balanced training with different objectives are all able to reduce bias, and the objective proposed by Lahoti et al. (2020) achieves the lowest TPR GAP. However, in terms of accuracyfairness trade-off, our proposed approach outperforms all other models, which is not surprising as it is designed to achieve better equal opportunity fairness. Based on these results, hereafter, we only report balanced training with our proposed EO objective (BTEO).

Main Results
We report results over the sentiment analysis and biography classification tasks in Table 2 , which also uses adversarial training but with multiple adversaries subject to an orthogonality constraint, and represents the current best baseline models.
On the MOJI dataset, compared to the STAN-  On the BIOS dataset, BTEO again leads to performance-fairness trade-offs that outperform the baseline methods. However, different to the MOJI dataset, BTEO does not further improve accuracy, improving fairness by 5.3% absolute at the cost of 2.9% accuracy.
In terms of training time, existing debiasing methods (esp. DADV on MOJI) incur a substantial overhead, while balanced training is much more frugal: around 1.3 times faster (because of the reduction in training data volume).
In addition to evaluating BTEO, we also combine GATE with BTEO, which achieves a better performance-fairness balance, as shown in Table 2. This is consistent with our argument that, rather than removing demographic information, properly used demographic factors can further reduce biases. Indeed, the BTEO +GATE consistently outperforms the current best baseline models model DADV on both datasets.
In Appendix F.1, we also show that BTEO can be combined with DADV and INLP, leading to better bias mitigation.

Gated Model
If the training dataset is imbalanced and contains spurious correlations between task labels and demographic attributes, a naively trained model will learn and possibly amplify dataset biases. The GATE model, with its explicit conditioning and group-specific encoding, will be particularly vulnerable to bias. Table 2 shows that, on both datasets, the GATE model increases the accuracy but amplifies bias (e.g., GAP of 65 on MOJI): as it uses demographic information directly to make predictions, it is highly vulnerable to bias in the training dataset.
Intuitively, the only objective of GATE training is standard cross-entropy loss, which has been shown to lead to bias amplification under imbalanced training without regularization. The gate components explicitly rely on demographic information, and thus become a strong indicator of predictions due to spurious correlations between the main task label and demographic labels in the training set.
Balanced training approaches act as regularizers in preventing the model from learning and amplifying spurious correlations in training.

Soft Averaging
Although the gated model naturally requires the demographic attribute at test time, we also evaluate a condition where this is not available. Instead, we take a Bayesian approach by evaluating p(y|x) = g p(g)p(y|g, x), where we can control the prior explicitly. For example, under a uniform demographic attribute prior, we simply average the predictions p(y|x, g) and p(y|x, ¬g). This Bayesian approach can be approximated by soft averaging, whereby the activation of all demographicspecific encoders are uniformly averaged inside the model, i.e., g i,j = 1 |G| , rather than selecting only one in the standard gated model (i.e., g i,: is 1-hot). When the protected attribute is observed at test time the soft averaging method may still prove useful, which we use as a means for fine-tuning the balance between accuracy and bias. Figure 3 shows an example for the prior fine-tuning. Specifically, we consider non-uniform encoder averaging conditioned on the gold protected attribute, g * . Let α and β denote to what extent the 1-hot labels are manipulated according to the value of g * as 0 and 1 respectively, leading to the soft labels α 1 − α and 1 − β β . I.e., the two specific encoders are weighted by either α and 1 − α, or 1 − β and β, respectively, according to the value of g * . Values of α, β < 0.5 mean the protected label is (softly) preserved, while values > 0.5 mean the label is flipped.
In cases where the model is biased towards or against a demographic group, it may be advantageous to use these two additional parameters to correct for this bias, by disproportionately using the other group's encoder.
We now employ the Bayesian "soft averaging" approach to gating, and mitigating bias at inference time. Note that this does not involve retraining the model, as the soft averaging happens at test time. Figure 3 shows accuracy and GAP results from tuning the coefficients on development data for the  Table 3: Results for balanced training methods on the BIOS test set. "RW" = instance reweighting; "DS" = dataset down-sampling; and "Size" = the number of instances in the training dataset. Bold = best trade-off.
basic GATE model. The results show that α = β = 0.5 is a reasonable default setting, however gains may be possible for non-uniform prior settings.
To demonstrate the power of adjusting these parameters, we take the trained GATE model, and then optimize α and β over the development set, and report the corresponding results on the test set. We select the parameter values that achieve the lowest development GAP, provided accuracy is above a threshold. 5 The results are reported in Table 2, under GATE soft RMS . On the MOJI dataset, our results show that GATE with soft averaging consistently outperforms the STANDARD and GATE models without balanced training. In terms of GAP, the model is substantially better than all other models, while remaining competitive in terms of accuracy. The BIOS dataset is noisier, meaning there are bigger discrepancies between the development and test datasets. However, we achieve a good performancefairness trade-off at a level comparable to the much more complex INLP and DADV models. Table 3 shows the results of the naively-trained MLP model ("STANDARD") and six balancedtraining methods, all based on the same architecture as STANDARD. Corpus down-sampling ("DS") removes instances from majority groups and thus leads to less training data and overall lower accuracy than instance reweighting ("RW").

Reweighting vs. Down-sampling
When using BD as the objective, both RW and DS perform similarly to the STANDARD model, as the overall gender distribution is quite balanced,  which can also be seen in the size of the training data for DS + BD. Both RW + JB and RW + EO reduce bias and performance, but RW + JB outperforms RW + EO in terms of the performancefairness trade-off, in that RW + JB achieves similar performance but substantially better fairness (6.6% absolute improvement in GAP). However, JB is not as effective as EO when combined with DS, due to the big drop in the volume of training data.

GATE vs. LARGE
Compared to STANDARD, GATE involves more model parameters, and an important question is whether the gains of GATE models are simply because of the larger parameter space.
To explore this question, we conduct experiments over MOJI by controlling the number of parameters of STANDARD and compare it with GATE models. Specifically, we employ a larger STANDARD model, namely LARGE, which has more hidden units within each hidden layer, leading to roughly the same number of parameters as the GATE model. Table 4 shows the results for STANDARD LARGE and BTEO with LARGE. Comparing STANDARD and BTEO with the LARGE versions, it can be seen that increasing the number of parameters does not increase performance or fairness.
Despite BTEO +GATE having fewer parameters than BTEO LARGE, it achieves a substantial improvement in terms of fairness, confirming our argument that GATE is superior to existing models.

Related Work
Fairness Much work on algorithmic fairness has focused on group fairness, i.e., disparities in error rates across groups defined by protected attributes, such as gender, age, or race. Many criteria have been proposed for group fairness, such as statistical parity (Dwork et al., 2012) and equal opportunity (Hardt et al., 2016). Broadly speaking, fairness can be classified into three categories: independence, separation, and sufficiency (Barocas et al., 2019), with the most recent work addressing separation criteria, i.e, potential correlations between main task labels and protected attributes. In the context of NLP, the best results have been achieved through protected information removal. Iterative nullspace projection (Ravfogel et al. (2020)) takes hidden representations and projects them onto the nullspace of the weights of a linear classifier for each protected attribute. The classifier training and projection are carried out over multiple iterations to more comprehensively remove protected information.

Mitigating bias
Another popular approach is adversarial training, which jointly optimizes the removal of sensitive information and main task performance, through the incorporation of adversarial discriminator(s) to identify protected attributes from the hidden repre-

Conclusions and Future Work
This paper proposed the adoption of balanced training approaches to mitigate bias, and demonstrated their effectiveness relative to existing methods, as well as their ability to further enhance existing methods. We also proposed a gated model based on demographic attributes as an input, and showed that while the simple version was highly biased, with a simple Bayesian extension at inference time, the method was highly effective at mitigating bias.
For future work, it is important to consider settings where there are multiple protected attributes, such as author age, gender, and ethnicity. A simple extension would be to treat G as being intersectional classes, defined as the Cartesian product of the multiple demographic groups. E.g., k binary groups would result in 2 k intersectional classes.

Ethical Considerations
This work aims to advance research on bias mitigation in NLP. Although our proposed method requires access to training datasets with protected attributes, this is the same data assumption that is made by other related work such as adversarial training and INLP, and our target is to make fair predictions at inference time. One limitation of methods in this area, including ours, is the difficulty of training and evaluating when we don't have access to demographic attributes. To avoid user harm, we only use attributes which users have self identified in our experiments. Moreover, our proposed method is able to make fairer predictions either with or without demographic information. All data in this study is publicly available and used under strict ethical guidelines.

Limitations
(1) We only investigate bias mitigation over English tasks. However, our method is naturally generalized to other languages.
(2) Many of debiasing methods assume that protected attributes are available in the dataset. While we acknowledge that protected attributes are normally unavailable, we follow previous work's setup and only use self-identified attributes in experiments. for future work is incorporating models that can debias with less protected labels in order to deal with real-world data, such as decoupled adversarial training (Han et al., 2021a) and adversarial instance reweighting (Lahoti et al., 2020).
(3) For both INLP and DADV, we follow experimental setup from the original papers. However, the fairlib (Han et al., 2022) -which is presented after this work was done -recently show that both methods can obtain better results with a larger budget for hyperparameter fine-tuning.

A.2 BIOS
Since the data is not directly available, in order to construct the dataset, we re-scrape the data with the scripts of Ravfogel et al. (2020), leading to a dataset with 396k biographies, which we randomly split into train (65%), dev (10%), and test (25%).

B Reproducibility B.1 Models
For INLP (Ravfogel et al., 2020), we take the fixed STANDARD model for the given dataset, and iteratively train a linear classifier and perform nullspace projection over the learned representation. For the other baseline models -ADV and DADV-we jointly train the adversarial discriminators and classifier. In order to ensure a fair comparison, we follow Han et al. (2021a) in using a model consisting of the same fixed-parameter encoder as ours followed by a trainable 3-layer MLP.

B.2 Hyperparameter Tuning
All approaches proposed in this paper share the same hyperparameters as the standard model. Hyperparameters are tuned using grid-search, in order to maximise accuracy for the standard model, and to minimise the fairness GAP for debiasing methods, subject to the accuracy exceeding a given threshold. The accuracy threshold is chosen to ensure the selected model achieves comparable performance to baseline methods, defined as up to 2% less than best baseline accuracy. Taking RW as an example, the best baseline accuracy on the BIOS development dataset is 75.7% and accordingly the (development) accuracy threshold is set to 73.7%; among models in the hyperparameter search space that exceed this threshold, we take the model with minimum GAP. We report test results for the selected models.
In terms of the baseline models, both DADV and INLP have additional hyperparameters: for DADV these are the weight of the adversarial loss, which controls the performance-fairness trade-off; the number of sub-adversaries; and the weight of the difference loss, to better remove demographic information; while INLP also has a trade-off hyperparameter, the number of null-space projection iterations, and other hyperparameters related to linear attackers and classifiers.
The trade-off hyperparameter makes such models more flexible in performing model selection. However, it also requires manual selection for better trade-offs, and different strategies have been introduced. For example, INLP manually selects the model at a iteration where the accuracy is minimally damaged while the fairness improves greatly. Similar manual selection for better trade-offs is also required for ADV and DADV, but the strategies proposed in the original papers are slightly different to one another, and are also task-specific.
In order to reproduce previous methods, we follow the original paper in setting the accuracy threshold, and then tuning hyperparameters for the best fairness.
For the ADV and DADV models, following the work of Han et al. (2021b), we tune extra hyperparameters separately, such as the trade-off hyperparameter, while using the same shared hyperparameters to the selected base models. Similarly, the number of iterations for the INLP model is tuned once other hyperparameters have been fixed.

B.3 Training Details
We conduct all our experiments on a Windows server with a 16-core CPU (AMD Ryzen Threadripper PRO 3955WX), two NVIDIA GeForce RTX 3090s with NVLink, and 256GB RAM.

B.3.1 MOJI
For all baseline models, we follow the method of Han et al. (2021b). Specifically, we train the STANDARD model for 100 epochs with the Adam optimizer (Kingma and Ba, 2015), learning rate of 3 × 10 −5 , and batch size of 1024. For ADV, the main model is jointly trained together with adversaries which are implemented as 3-layer MLP, and the weight of adversarial loss is 0.8. For each iteration (epoch) of the main model, an adversary is trained for 60 epochs, keeping the checkpoint model that performs best on the dev set. Three sub-adversaries are employed by the DADV, with the difference losss weight of 10 3.7 . For INLP, logistic regression models are used for both identifying null-space to the demographic information at each iteration, and making the final predictions given debiased hidden representations. Since the number of iterations in INLP is highly affected by the random seed at each run, we re-select it at each iteration.
As for our models, the DS model is trained with the learning rate of 10 −5 and batch size of 512; the RW is trained with the learning rate of 10 −4 and batch size of 1024; and the GATE is trained with the the set of hyperparameters to the base model.

B.3.2 BIOS
Models are trained with similar hyperparameters as models on the MOJI dataset. We thus only report main differences for each of them: the STANDARD model is trained with the batch size of 512 and learning rate of 3 × 10 3 ; DS models are trained with the batch size of 128 and learning rate of 10 −3 , and RW models are trained with the batch of 256 and learning rate of 3 × 10 −5 .
We train the ADV model with the adversarial loss weight of 10 −2.3 , learning rare for adversarial training of 10 −1 , learning rate of 10 −3 , and batch size of 128. The DADV is trained with same setting as the ADV, excepting the difference loss weight of 10 2 . For details of the assignment of other hyperparameters and hyperparameter searching space, refer to Supplementary Materials.

C The calculation of DTO
For ease of comparison between approaches, we introduce 'distance to the optimum' (DTO), a single metric to incorporate accuracy and GAP into a single figure of merit, which is calculated by: (1) converting GAP to 1 − GAP (denoted as fairness; higher is better); (2) normalizing each of accuracy and fairness, by dividing by the best result for the given dataset (i.e., highest accuracy and fairness); and (3) calculating the Euclidean distance to the point (1, 1), which represents the hypothetical system which achieves highest accuracy and fairness for the dataset. Lower is better for this statistic, with minimum 0.
We calculate DTO based on all results shown in Table 7. Taking the DAdv model on the Moji dataset for example, the trade-off is calculated as follows: 1. Find the best accuracy and fairness (1-GAP) separately; i.e., 74.9 (GATE + RW) and 92.9 (GATE soft RMS ), resp. 2. Normalize the accuracy and fairness metric of DADV, resulting in 0.995 = 74.5 74.9 and 0.877 = 81.5 92.9 .

D Training time estimation
Given that the training time is affected by factors such as batch size, hidden size, and learning rate, to perform a fair comparison between different models, we estimate the training time of a model based on hyperparameter tuning results, over a shared search space of base hyperparameters (i.e., the hyperparameters related to the standard model), with any other approach-specific hyperparameters fixed.

E Mapping of previous objectives
As for the jointly balance, Lahoti et al. (2020) state that "In addition to vanilla inverse probability weighting (IPW), we also report results for an IPW variant with inverse probabilities computed jointly over protected-features and class-label reported as IPW(S+Y)". It clearly shows that instance from group g with class y is weighted by 1 ny,g . Adding this weight to the unweighted loss function leads to y g ny,g n 1 ny,g L y,g , which exactly the objective that is shown in our paper.
In terms of the conditionally balance, in the section 'Alternative Data Splits' of Wang et al. (2019) Table 7: Results over the sentiment analysis (MOJI) and biography classification (BIOS) tasks. Trade-offs are measured by the normalized Euclidean distance between each model and the ideal model, and lower is better. Bold = best trade-off within category. Training time is reported relative to STANDARD, which takes 35 secs and 16 mins for MOJI and BIOS, respectively. they obtain the dataset by resampling such that the number of occurrences of men with label y and of women with label y is close, i.e., the size of different demographic groups are almost identical for each class-label. This is equivalent to assign equal weights to difference demographic groups within each class-label, i.e., y g ny,g n ny, * ny,g L y,g . Finally, for the balanced demographics, Zhao et al. (2018) balance the distribution of demographic groups which reweights instance of a demographic group inversely to its proportion, leading a weight n n * ,g . As a result, the final objective is y g ny,g n n n * ,g L y,g .

F Further extensions F.1 Combining balanced training with benchmark methods
The baseline methods INLP and DADV as presented above were used in a manner consistent with their original formulation, i.e., without balanced training. An important question is whether balanced training might also benefit these methods. It is trivial to combine downsampling with INLP and DADV, as the method simply prunes the training dataset, but does not impact the training objective. To combine instance reweighting with DADV, we modify the training objective such that the cross-entropy term is scaled byp −1 , while leaving the adversarial term unmodified, i.e., solve for min M max A (x i ,y i ,g i )∈Dp −1 X (y i ,ŷ i ) − λ adv X (g,ĝ). For INLP, we simply train a BTEO model, and then iteratively perform INLP linear model training and nullspace projection over the learned representations.
Results are presented in the final section of Table 7 ("Combination"), and indicate that the combined methods appreciably outperform both the standalone demographic removal methods and balanced training approaches, without extra training time cost. That is, demographic information removal and balanced training appear to be complementary.

F.2 Balancing toward anti-stereotyping
As shown in Table 2, even with DS or RW balancing, the model still shows biases in its predictions. We conduct preliminary experiments on MOJI with RW and DS, while controlling for stereotyping skew in training using values for 0.8 to 0.2. In standard rebalancing we use as target 0.5, which describes a balanced situation. A larger skew > 0.5 will amplifying stereotyping, and < 0.5 describes a different type of stereotyping operating in the opposite direction. Balancing towards a 0.4 training skew leads to the best test results, with an accuracy of 71.7% and GAP of 11.8% for DS, and accuracy of 74.5% and GAP of 11.3% for RW. Comparing to the corresponding values in Table 2 (rows Balance DS and RW, for MOJI), both results show a substantial reduction in GAP.
This idea is related to existing reweighting approaches in long-tail learning. For example, Cui et al. (2019) infer the effective number of samples which group each instance with its neighbours within a small region instead of using all data points, and reweight the loss of each class inversely proportional to the effective number of samples. We leave this further exploration of this line of research to future work.
We also experiment with GATE +RW and GATE +DS with a 0.4 training skew, however, the gated model does not show the same behaviour, as it just amplifies the training biases. This implies that, for the gated model, balanced training can help remove spurious correlations between protected attributes and main task labels, which is similar in nature to the effects of adversarial training.

G Comparison with MoE
One line of work that is similar to our gated model is mixture of experts (Ma et al., 2018;Fedus et al., 2021). Technically, the gated model is similar to the MoE model in the sense that an expert can be largely aligned with a group-specific encoder in our model. However, there are several key differences: (1) instead of making independent predictions by each expert, our group-invariant encoder acknowledges the shared patterns across demographic groups; (2) MoE employs an extra softmax gating network to mix experts' predictions, while our method does an argmax based on group labels; and (3) we use the group-specific information jointly together with the group-invariant encoder' outputs for making the final predictions while the MoE model has one output layer for each expert.