{'fraction_non_alphanumeric': 0.04001710654936461, 'fraction_numerical': 0.011852394916911046, 'mean_word_length': 4.919891500904159, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Upstream Mitigation Is 
 Not
 All You Need: Testing the Bias Transfer Hypothesis in Pre-Trained Language Models

A few large, homogenous, pre-trained models undergird many machine learning systems — and often, these models contain harmful stereotypes learned from the internet. We investigate the bias transfer hypothesis: the theory that social biases (such as stereotypes) internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning. For two classification tasks, we find that reducing intrinsic bias with controlled interventions before fine-tuning does little to mitigate the classifier’s discriminatory behavior after fine-tuning. Regression analysis suggests that downstream disparities are better explained by biases in the fine-tuning dataset. Still, pre-training plays a role: simple alterations to co-occurrence rates in the fine-tuning dataset are ineffective when the model has been pre-trained. Our results encourage practitioners to focus more on dataset quality and context-specific harms.


Introduction
Large language models (LLMs) and other massively pre-trained "foundation" models are powerful tools for task-specific machine learning (Bommasani et al., 2021). Models pre-trained by wellresourced organizations can easily adapt to a wide variety of downstream tasks in a process called finetuning. But massive pre-training datasets and increasingly homogeneous model design come with well-known, immediate social risks beyond the financial and environmental costs (Strubell et al., 2019;Bender et al., 2021).
Despite these risks, no research has investigated the extent to which downstream systems inherit social biases from pre-trained models. 1 Many stud-ies warn that increasing intrinsic bias upstream may lead to an increased risk of downstream harms (Bolukbasi et al., 2016;Caliskan et al., 2017). This hypothesis, which we call the Bias Transfer Hypothesis, holds that stereotypes and other biased associations in a pre-trained model are transferred to post-fine-tuning downstream tasks, where they can cause further, task-specific harms. A weaker version of this hypothesis holds that downstream harms are at least mostly determined by the pretrained model (Bommasani et al., 2021).
In the pre-training paradigm, the extent to which the bias transfer hypothesis holds will determine the most effective strategies for responsible design. In the cases we study, reducing upstream bias does little to change downstream behavior. Still, there is hope: instead, developers can carefully curate the fine-tuning dataset, checking for harms in context.
We test the bias transfer hypothesis on two classification tasks with previously demonstrated performance disparities: occupation classification of online biographies (De-Arteaga et al., 2019) and toxicity classification of Wikipedia Talks comments (Dixon et al., 2018). We investigate whether reducing or exacerbating intrinsic biases encoded by RoBERTa (Liu et al., 2019) decreases or increases the severity of downstream, extrinsic harms (Figure 1). We find that the bias transfer hypothesis describes only part of the interplay between pretraining biases and harms after fine-tuning: • Systematically manipulating upstream bias has little impact on downstream disparity, especially for the most-harmed groups.
• With a regression analysis, we find that most variation in downstream bias can be explained by bias in the fine-tuning dataset (proxied by cooccurrence rates).
• Altering associations in the fine-tuning dataset can sometimes change downstream behavior, but only when the model is not pre-trained.
Without absolving LLMs or their owners of representational harms intrinsic to pre-trained models, our results encourage practitioners and application stakeholders to focus more on dataset quality and context-specific harm identification and reduction. that result in representational or allocational harms to historically marginalized social groups (Blodgett et al., 2020).

Related Work
Little prior work directly tests the bias transfer hypothesis. The closest example of this phenomena is the "blood diamond" effect (Birhane and Prabhu, 2021), in which stereotyping and denigration in the pre-training corpora pervade subsequently generated images and language even before the fine-tuning stage (Steed and Caliskan, 2021). Still, it is unclear to what extent undesirable values encoded in pre-training datasets or benchmarks-such as Wikipedia or ImageNetinduce task-specific harms after fine-tuning (Barocas et al., 2019). Some work explores the consistency of intrinsic and extrinsic bias metrics: Goldfarb-Tarrant et al. (2021) find that intrinsic and extrinsic metrics are not reliably correlated for static embeddings like word2vec. We focus instead on stateof-the-art transformer-based LLMs-the subject of intense ethical debate (Bender et al., 2021;Bommasani et al., 2021)-which construct contextual, rather than static, embeddings. Contextual embeddings-token encodings that are conditional on other, nearby tokens-pose an ongoing challenge for intrinsic bias measurement (May et al., 2019;Kurita et al., 2019;Guo and Caliskan, 2021) and bias mitigation (Liang et al., 2020). We find that intrinsic and extrinsic metrics are correlated for the typical LLM-but that the correlation is mostly explained by biases in the fine-tuning dataset.
Other research tests the possibility that upstream mitigation could universally prevent downstream harm. Jin et al. (2021) show that an intermediate, bias-mitigating fine-tuning step can help reduce bias in later tasks. Likewise, Solaiman and Dennison (2021) propose fine-tuning on carefully curated "values-targeted" datasets to reduce toxic GPT-3 behavior. Our results tend to corroborate these methods: we find that the fine-tuning process can to some extent overwrite the biases present in the original pre-trained model. A recent post-hoc mitigation technique, on the other hand, debiases contextual embeddings before fine-tuning (Liang et al., 2020). Our results imply that while this type of debiasing may help with representational harms upstream, it is less successful for reducing harms downstream.

Methods
To empirically evaluate the bias transfer hypothesis, we examine the relationship between upstream bias and downstream bias for two tasks. We track how this relationship changes under various controlled interventions on the model weights or the fine-tuning dataset.

Occupation Classification
The goal of occupation classification is to predict someone's occupation from their online biography. We fine-tune with the BIOS dataset scraped from Common Crawl (De-Arteaga et al., 2019), which includes over 400,000 online biographies belonging to 28 common occupations. Since selfidentified gender was not collected, we will refer instead to the pronouns used in each biography (each biography uses either he/him or she/her pronouns). Following De-Arteaga et al. (2019), we use the "scrubbed" version of the dataset-in which all the identifying pronouns have been removed-to measure just the effects of proxy words (e.g. "mother") and avoid overfitting on pronouns directly.
Downstream Bias.-Biographies with she/her pronouns are less frequently classified as belonging to certain traditionally male-dominated professions-such as "surgeon"-which could result in lower recruitment or callback rates for job candidates if the classifier is used by an employer. The empirical true positive rate (TPR) estimates the likelihood that the classifier correctly identifies a person's occupation from their biography.
We follow previous work (De-Arteaga et al., 2019) in measuring downstream bias via the empirical true positive rate (TPR) gap between biographies using each set of pronouns. First, define where g is a set of pronouns and y is an occupation. Y andŶ represent the true and predicted occupation, respectively. Then the TPR bias (TPB) is TPB y = TPR y,she/her TPR y,he/him . For example, the classifier correctly predicts "surgeon" for he/him biographies much more often than for she/her biographies, so the TPR ratio for the "surgeon" occupation is low (see Appendix A).
Upstream Bias.-We adapt Kurita et al. (2019)'s pronoun ranking test to the 28 occupations in the BIOS dataset. Kurita et al. (2019) measure the encoded association of he/him and she/her pronouns by the difference in log probability scores between pronouns appearing in templates of the form {pronoun} is a(n) {occupation}. We augment this approach with 5 career-related templates proposed by Bartl et al. (2020) (see Appendix A). Formally, given a template sequence x y,g filled in with occupation y and pronoun g, we compute p y,g = P(x y,g ). As a baseline, we also mask the occupation and compute the prior probability ⇡ y,g = P(x ⇡ y,g ). The pronoun ranking bias (PRB) for this template is the difference in log probabilities: PRB y = log p y,she/her ⇡ y,she/her log p y,he/him ⇡ y,he/him . (2)

Toxicity Classification
For toxicity classification, we use the WIKI dataset, which consists of just under 130,000 comments from the online forum Wikipedia Talks Pages (Dixon et al., 2018). The goal of the task is to predict whether each comment is toxic. Each comment has been labeled as toxic or non-toxic by a human annotator, where a toxic comment is a "rude, disrespectful, or unreasonable comment that is likely to make you leave the discussion" (Dixon et al., 2018). Following Dixon et al. (2018), we focus on 50 terms referring to people of certain genders, races, ethnicities, sexualities, and religions. Downstream (Extrinsic) Bias.-Mentions of certain identity groups-such as "queer"-are more likely to be flagged for toxic content, which could result in certain communities being systematically censored or left unprotected if an online platform uses the classifier. The classifier's empirical false positive rate (FPR) estimates its likelihood to falsely flag a non-toxic comment as toxic. The FPR corresponds to the risk of censoring inclusive speech or de-platforming individuals who often mention marginalized groups.
Following Dixon et al. (2018), we express the classifier's bias against comments or commenters harmlessly mentioning an identity term as the FPR bias (FPB).
where i is an identity term and T = 1 if the comment was deemed toxic by a human annotator. Upstream Bias.-Following Hutchinson et al. (2020), we measure upstream bias via sentiment associations. We construct a set of templates of the form {identity} {person} is [MASK], where identities are the identity terms from Dixon et al. (2018) (e.g. "gay" or "Muslim") and the person phrases include "a person," "my sibling," and other relations. We predict the top-20 likely tokens for the "[MASK]" position (e.g., "awesome" or "dangerous"). Using a pre-trained RoBERTA sentiment classifier trained on the TweetEval benchmark (Barbieri et al., 2020), we then measure the average negative sentiment score of the predicted tokens. The model's bias is the magnitude of negative association with each identity term.
RoBERTa sometimes suggests terms which refer back to the target identity group. To mitigate this effect, we drop any predicted tokens that match the 50 identity terms (e.g. "Latino") from Dixon et al. (2018), but we are likely missing other confounding adjectives (e.g. "Spanish"). We suspect this confounding is minimal: we achieve similar results with an alternative ranking-based bias metric (see Appendix C.2).

Experiments
We measure changes in upstream and downstream bias subject to the following interventions ( Fig. 1): • No pre-training. To control for the effects of pre-training, we test randomly initialized versions of both models that have not been pretrained. We average over 10 trials.
• Random perturbations. We instantiate a pretrained model and then add random noise e to every weight in the embedding matrix. We try both uniform noise u ⇠ Unif( c, c) and Gaussian noise z ⇠ N (0, 2 ), varying c and 2 . The final noise-added matrix is clipped so that its range does not exceed that of the original matrix.
• Bias mitigation. We apply the SENTDEBIAS algorithm to de-bias embeddings at the word-level (Liang et al., 2020). SENTDEBIAS estimates a bias subspace V with principal component analysis, then computes debiased word embeddingŝ h = h P k j=1 hh, v j iv j by subtracting the projection of h onto V. We add the multiplier to add or remove bias to various degreesstandard SENTDEBIAS uses = 1.0.
• Re-balancing and scrubbing. For BIOS, we re-balance the fine-tuning dataset by undersampling biographies with the prevalent pronoun in each occupation. For WIKI, we randomly remove from the fine-tuning dataset ↵ percent of comments mentioning each identity term.

Upstream variations have little impact on downstream bias.
Our goal is to test the bias transfer hypothesis, which holds that upstream bias is transferred through fine-tuning to downstream models. By this view, we would expect changes to the pretrained model to also change the distribution of downstream bias-but we find that for both tasks, downstream bias is largely invariant to upstream interventions. Figure 2 summarizes the similarity of biases before and after each randomized event. Though randomizing the model weights significantly reduces the mean and variance of upstream bias, the distribution of downstream bias changes very little. 4 For example, RoBERTa exhibits the same disparities in performance after fine-tuning regardless of whether the base model was pre-trained. Likewise, although the SENTDEBIAS mitigation method reduces pronoun ranking (upstream) bias as intended, we detect roughly the same downstream biases no matter the level of mitigation applied ( Figure 3). For example, in the BIOS task, surgeons with he/him pronouns are still 1.3 times more likely to have their biographies correctly classified than their she/her counterparts.
There is one notable exception to this trend: for the WIKI task, adding noise (uniform or Gaussian) to the pre-trained model's embedding matrix or not pre-training the model yields a modest reduction in median bias ( Figure 2). As upstream bias shifts towards zero, downstream bias also moves marginally towards zero. Still, the largest biases (e.g., against the term "gay") do not decrease and may even increase after randomization.

Most downstream bias is explained by the fine-tuning step.
Though the results in the preceding section suggest that there is no clear or consistent correspondence between changes in upstream bias and changes in downstream bias, there is still a noticeable correlation between baseline upstream and downstream bias (Pearson's ⇢ = 0.43, p = 0.022 for BIOS, ⇢ = 0.59, p < 10 5 for WIKI-see Appendix A). There is an important third variable that helps explain this correlation: cultural artifacts ingrained in both the pre-training and finetuning datasets. 5 RoBERTa learns these artifacts through co-occurrences and other associations between words in both sets of corpora.
To test this explanation, we conduct a simple regression analysis across interventions ( Figure 1) and evaluation templates. We estimate log TPB m,y = 0 + 1 PRB m,y,s + 2 ⇡ y +f s +c m .
(4) for model treatment m, occupation y, and pronoun ranking template s. TPB is the TPR bias (downstream bias) from Eq. 1; PRB is the pronoun ranking bias (upstream bias) from Eq. 2; f s and c m are dummy variables (for ordinary least squares) or fixed effects to capture heterogeneous effects between templates and models (such as variations in overall embedding quality). We control for statistical "dataset bias" with ⇡, the prevalence of "she/her" biographies within each occupation y in the fine-tuning data.
We find that the "dataset bias" in the fine-tuning stage explains most of the correlation between upstream and downstream bias. Under the strong bias transfer hypothesis, we would expect the coefficient on upstream bias 1 to be statistically significant and greater in magnitude than the coefficient : Log TPR bias per occupation after scaled SENTDEBIAS on the BIOS task. Mitigation significantly reduces pronoun ranking (upstream) bias compared to base RoBERTa (top); but even when upstream bias decreases, the TPR ratio (downstream bias) remains mostly constant (bottom). The distribution of downstream bias without any mitigation is almost perfectly correlated with the distribution at = 50 (Pearson's ⇢ = 0.96, p < 0.01).
the opposite is true: fine-tuning dataset bias has a larger effect than upstream bias. Figure 4 reports the coefficient estimates for these two variables.
(See Appendix C.1 for all estimates, standard errors, assumptions and additional specifications.) In the BIOS task, a large decrease in upstream bias corresponds to a small but statistically significant increase in downstream bias. On average, a reduction of 0.3 to the log likelihood gap-equivalent to the reduction in bias towards nurses after upstream mitigation-corresponds to a 0.5% increase in the TPR ratio. Almost all the downstream bias in the BIOS task is explained by dataset bias instead: a 10% increase in the prevalence of she/her pronouns within an occupation corresponds to a much larger 6.5% increase in the TPR ratio.
In the WIKI task, upstream bias has a more noticeable effect-but the effect of dataset bias is still much larger. The regression takes the same form as Eq. 4, where downstream bias is FPR bias (Eq. 3), upstream bias is negative sentiment, and ⇡ i is the proportion of toxic mentions of identity i. We additionally control for the prevalence of each identity term and the average length of toxic mentions of each identity term-longer comments are less likely to result in erroneous screening (Appendix C.1).
As in the previous regression, dataset bias explains more of the variation in downstream bias than does upstream bias. On average, a large increase in average negative sentiment against a given identity term (e.g. 0.1, one standard deviation) corresponds to only a modest 3.7% increase in FPR. In comparison, only a 10% increase in the prevalence of toxic mentions of an identity corresponds to an even larger 6.3% increase in FPR.
We also check that intrinsic downstream bias also changes due to fine-tuning. We measure intrinsic bias again after fine-tuning and regress on downstream intrinsic bias instead of downstream extrinsic bias (Eq. 4). The results are consistent: after controlling for the overall increase in log likelihood, the effect of upstream intrinsic bias on downstream intrinsic bias is explained almost entirely by fine-tuning dataset bias (Appendix C.1).

Re-sampling and re-scrubbing has little effect on downstream behavior.
Given the strong relation between our proxies for dataset bias and downstream bias, we test whether manipulating these proxies admits some control over downstream bias. For example, were the finetuning dataset to include exactly as many she/her nurse biographies as he/him, would the model still exhibit biased performance on that occupation? Our findings suggest not. No matter the amount of re-sampling, downstream bias remains relatively stable for pre-trained RoBERTa. The distributions of downstream bias with and without re-balancing are almost perfectly correlated (Pearson's ⇢ = 0.94, p < 0.01-see Appendix B.1). Though co-occurrence statistics help to explain downstream bias, they are still only proxies for dataset bias. Directly altering these statistics via re-sampling the dataset does not alter the sentencelevel context in which the words are used.
Based on this result, we also try completely removing mentions of identity terms. Scrubbing mentions of identity terms-in all comments or only in toxic comments-appears to reduce bias only when the model is not pre-trained and all mentions of the term are scrubbed ( Figure 5). For a pre-trained model trained on scrubbed data, a 10% decrease in mentions of an identity term corresponds to a 7.2%

Upstream bias
Avg. negative sentiment Figure 4: Effect of upstream bias vs. fine-tuning dataset bias on downstream bias, controlling for model & template fixed effects. Bars depict heteroskedasticity-consistent standard errors. Statistically insignificant (p < 0.01) coefficients are hollow. For both tasks, reduction in fine-tuning dataset bias corresponds to a greater alteration to downstream bias than an equivalent reduction (accounting for scale) in upstream bias.
decrease in FPR. We speculate that RoBERTa relies on its high quality feature embeddings to learn proxy biases about identity terms based on the way they are used in the pre-training corpora. For example, our model classifies a sequence containing only the term "gay" as toxic without any context. If a term like "gay" is often used pejoratively on the web, RoBERTa is likely to infer that sentences including "gay" are toxic even if the term never appears in the fine-tuning dataset.
But when the upstream model is not pre-trained, the fine-tuned model has no such prejudices. In this case, removing all mentions of identity results in a distribution of bias entirely uncorrelated with the control (Pearson's ⇢ = 0.09, p > 0.1). Notably, though, even a small number of mentions of an identity term like "gay" in the fine-tuning dataset are enough for a randomly initialized model to exhibit the same biases as the pre-trained model ( Figure 5).

Limitations
Our approach comes with several limitations. First, our results may not generalize to all tasksespecially non-classification tasks-or all kinds of bias (e.g., bias against AAVE or non-English speakers). Also, while similar studies of bias have been successfully applied to vision transformers (Steed and Caliskan, 2021; Srinivasan and Uchino, 2021), our results may vary for substrates other than English language.
Second, Goldfarb-Tarrant et al. (2021) conclude that the lack of correlation between intrinsic bias indicators and downstream bias is because some embedding bias metrics are unsuitable for measuring model bias. To ensure our intrinsic and extrinsic metrics measure the same construct, we chose upstream indicators that correlate with realworld occupation statistics (Caliskan et al., 2017;Kurita et al., 2019). Pronoun ranking in particular may be more reliable for transformer models than Third, there may be other explanations for the relationship between upstream and downstream bias: for example, decreasing the magnitude of upstream bias often requires a reduction in model accuracy, though we attempt to control for between-model variation with fixed effects and other controls. Alternate regression specifications included in Appendix C.1 show how our results change with the inclusion of controls.

Conclusion
Our results offer several points of guidance to organizations training and distributing LLMs and the practitioners applying them: • Attenuating downstream bias via upstream interventions-including embedding-space bias mitigation-is mostly futile in the cases we study and may be fruitless in similar settings.
• For a typical pre-trained model trained for the tasks we study, the fine-tuning dataset plays a much larger role than upstream bias in determining downstream harms.
• Still, simply modulating co-occurrence statistics (e.g., by scrubbing harmful mentions of certain identities) is not sufficient. Task framing, design, and data quality are also very important for preventing harm.
• If a model is pre-trained, it may be more resistant to scrubbing, re-balancing, and other simple modulations of the fine-tuning dataset.
But, our results also corroborate a nascent, somewhat optimistic view of pre-training bias. LLMs' intrinsic biases are harmful even before downstream applications, and correcting those biases is not guaranteed to prevent downstream harms. Increased emphasis on the role of fine-tuning dataset bias offers an opportunity for practitioners to shift to more careful, quality-focused and context-aware approach to NLP applications (Zhu et al., 2018;Scheuerman et al., 2021).

Ethical Considerations
This study navigates several difficult ethical issues in NLP ethics research. First, unlike prior work, we do not claim to measure gender biases-only biases related to someone's choice of personal pronouns. However, our dataset is limited to the English "he/him" and "she/her," so our results do not capture biases against other pronouns. Our study is also very Western-centric: we study only English models/datasets and test for biases considered normatively pressing in Western research. Second, our training data (including pre-training datasets), was almost entirely scraped from internet users without compensation or explicit consent. To avoid exploiting these users further, we only used already-scraped data and replicated alreadyexisting classifiers, and we do not release these data or classifiers publicly. Finally, the models we trained exhibit toxic, offensive behavior. These models and datasets are intended only for studying bias and simulating harms and, as our results show, should not be deployed or applied to any other data except for this purpose.

A Descriptive Statistics
BIOS.-Biographies include the 28-most frequent occupations according to the BLS Standard Occupation Classification system. 6 See Figure 6 for a full list of occupations and the prevalence of each set of pronouns. Upstream bias (measured with pronoun ranking) is depicted in Figure 7. Table 1 Gives the full list of templates used for testing. Traditionally female occupations (e.g., "nurse") are generally biased towards "she/her," with some exceptions (e.g., "software engineer"). Downstream bias is similarly distributed- Figure 9 depicts the relationship between upstream and downstream bias, which is generally linear (Pearson's ⇢ = 0.43, p < 0.05). There are noticeable outliers (e.g., "surgeon") for which real-world harms could be especially acute.
WIKI.-Identity terms occur relatively infrequently in the dataset and some terms appear in toxic comments more often than others. See Figure 8 for a full list of identity terms tested and their prevalence in toxic and non-toxic comments.
Upstream and downstream bias have an even stronger relationship for WIKI than for BIOS (⇢ = 0.59, p < 0.01). The terms "homosexual" and "gay" have especially high biases both upstream and downstream, likely because of their use as common pejoratives on certain internet spaces represented in the pre-training and fine-tuning datasets.

B.1 Partial Interventions
This section gives the full results for each partial intervention (e.g., scrubbing only half of the 6 https://www.bls.gov/soc/  toxic mentions, rather than all of them). Figure 10 shows downstream bias at each level of pronoun re-balancing. To measure the effects of different levels of gender balance, we either randomly oversample the minority group to close ↵ percent of the gap between the majority and minority pronouns, randomly under-sample the majority group to close percent of the gap, or some combination of both. Figure 11 shows the full set of partial scrubbing interventions tested -notably, there is a slight convergence to zero in downstream bias when the model is not pre-trained.

B.2 Correlation Tests
To quantify the effect of our experimental interventions on upstream and downstream bias, we compared the distribution of bias after each in-  tervention to the distribution of bias after an unmodified pre-trained model. We tested for statistical correlation between these two distributions with both Pearson's correlation coefficient ⇢ and Kendall's correlation coefficient ⌧ . For Pearson's, we assume that the two distributions are approximately normally distributed. This assumptions seems reasonable because our samples are not too small (N = 28 and N = 50 for BIOS and WIKI, respectively), but a Shapiro-Wilk test of normality shows that downstream bias for both tasks is likely non-normal (W = 0.81 and W = 0.67 for TPR ratio and FPR ratio respectively, p < 0.01). So, we also compute Kendall's correlation coefficient ⌧ , which is a nonparametric test of ordinal association. The results are similar in magnitude and significance (Table 2).

C Full Regression Results
Tables 3 and 4 report the full set of coefficient estimates used to generate Figure 4 and the effects described in the paper. We use HC3 heteroskedasticity-consistent standard errors. 7 The Variance Inflation Factor (VIF) for the covariates are all less than 2.5 for an unmodified pre-trained model for both tasks (a sign that multicollinearity may not be too severe).
The fixed effects regression requires a few assumptions for unbiased, normally-converging estimates. First, we assume that the error is uncorrelated with every covariate (i.e., there are no omitted variables; we discuss this possibility in the limitations section). Second, we assume that the samples are independent and identically distributed (independence is assured by our experimental setup, which varies one factor at a time). Third, we assume that large outliers are unlikely (evident from the distribution plots presented).

C.1 Additional Specifications
We tested several regression specifications on just the unmodified, pre-trained model (Tables 5 and 6). For BIOS, note that the direct and indirect (after controlling for dataset bias) effects of upstream bias on downstream bias have opposite signs. The change is the effect of including dataset bias, a colinear confounder, in the regression. Confounders 7 For a simple OLS specification on WIKI, the Breusch-Pagan test rejects the hypothesis that our errors are homoskedastic with BP = 27.039, p < 10 3 . For BIOS, the hypothesis is not rejected (BP = 5.033, p = 0.41).
can be interpreted as "explaining" the relationship between the independent (upstream) and dependent (downstream) variables (MacKinnon et al., 2000).
To test whether the effect observed is mediated by a change in the model's weights, also include an estimate of the effect of upstream intrinsic bias (e.g., from pronoun ranking) on downstream intrinsic bias (intrinsic bias, measured after fine-tuning). We control for the overall increase in log likelihood by including in the regression the difference in log likelihood of the neutral pronouns "they/them" before and after fine-tuning. We find that a similar relationship holds between upstream bias and intrinsic bias downstream as holds between upstream bias and extrinsic bias downstream, suggesting that the model's internal representations change in concert with its downstream behavior.

C.2 Identity Ranking -Robustness Check
Because of the limitations of the sentiment-based approach, we check the robustness of our results with an identity ranking approach based on pronoun ranking. Included in the Dixon et al. (2018) study of toxicity classification bias is an extensive evaluation set composed of 89,000 templates such as "[IDENTITY] is [ATTRIBUTE]," where the attributes include both positive (for non-toxic examples) and extremely negative words (for toxic  Table 2: Correlation between bias distributions before and after each intervention. Statistically insignificant correlation coefficients indicate bias has changed drastically (red). Notably, downstream bias is correlated with the control to some extent for every intervention except for scrubbing and not pre-training. Pearson's correlation coefficient ⇢ measure of correlation strength and direction; Kendall's ⌧ is a measure of ordinal correlation. Randomized interventions (e.g., not pre-training, adding noise) tend to re-order the bias distribution more than others, indicated by a lower ⌧ .    examples). Templates of other forms are not included to reduce computation time. 8 For each of these templates, we mask the identity term and compute the log probability score as described in §3.2. The model's bias is described by the difference between the average log probability scores for the toxic templates and the non-toxic templates for each identity term. For the regressions (Tables 7 and 8), the templates are not paired, so we average first across toxic and non-toxic templates, then calculate the ratio between the two. The relative size and statistical significance of the coefficients are the same as for the negative sentiment approach, suggesting the negative sentiment metric is robust for our purposes despite its limitations.

D Replication
We provide our full results (upstream and downstream bias for every intervention, for each task) and the scripts used to analyse them. We are not allowed to release the source code used to train our models and measure bias, but we include additional details on our implementation to help others understand and replicate our results.