{'fraction_non_alphanumeric': 0.03305623948709198, 'fraction_numerical': 0.012359523171058725, 'mean_word_length': 5.2333991794560095, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hate Speech Classifiers Learn Human-Like Social Stereotypes

Social stereotypes negatively impact individuals' judgements about different groups and may have a critical role in how people understand language directed toward minority social groups. Here, we assess the role of social stereotypes in the automated detection of hateful language by examining the relation between individual annotator biases and erroneous classification of texts by hate speech classifiers. Specifically, in Study 1 we investigate the impact of novice annotators' stereotypes on their hate-speech-annotation behavior. In Study 2 we examine the effect of language-embedded stereotypes on expert annotators' aggregated judgements in a large annotated corpus. Finally, in Study 3 we demonstrate how language-embedded stereotypes are associated with systematic prediction errors in a neural-network hate speech classifier. Our results demonstrate that hate speech classifiers learn human-like biases which can further perpetuate social inequalities when propagated at scale. This framework, combining social psychological and computational linguistic methods, provides insights into additional sources of bias in hate speech moderation, informing ongoing debates regarding fairness in machine learning.

can impact a classifier's evaluation of a sentence about disability (Hutchinson et al., 2020).
Second, individual-level biases of annotators can impact the classifier in stereotypical directions. For example, a piece of rhetoric about disability can be analyzed and labeled differently depending upon the social biases of the annotators. Although previous research has documented stereotypes in language models (Garg et al., 2018;Bolukbasi et al., 2016;Manzini et al., 2019;Swinger et al., 2019;Charlesworth et al., 2021), the impact of annotators' own biases on training data and models remains largely unknown. This gap in our understanding of the effect of human annotation on biased NLP models is the focus of this work.
In this paper, we investigate how hate speech annotation is impacted by social stereotypes and whether, as a result, hate speech classifiers trained on human annotations show human-like biases in their predictions. Study 1 investigates the role of people's social stereotypes in their hate speech annotation behavior in a large, nationally stratified sample in the US. Study 2 explores the impact of social stereotypes embedded in language on aggregated behavior of expert annotators in a large corpus of hate speech, which is used for training hate speech classifiers. Finally, Study 3 demonstrates how language-embedded stereotypes are associated with biased predictions in a neural network model for hate speech detection. Before presenting our studies, we briefly review the social psychological literature on social stereotyping and discuss the theoretical framework on which our analyses are based.

Social Stereotyping
A comprehensive evaluation of human-like biases in hate speech classification needs to be grounded in social psychological theories of prejudice and stereotypes, in addition to how they are manifested in language (Blodgett et al., 2020;Kiritchenko et al., 2020).
Stereotyping is a cognitive bias, deeply rooted in human nature (Cuddy et al., 2009) and omnipresent in everyday life. Stereotyping affords humans with the capacity to promptly assess whether an outgroup is a threat or not. Along with other cognitive biases, stereotyping impacts how individuals create their subjective social reality as a basis for social judgements and behaviors (Greifeneder et al., 2017). Stereotypes are often studied in terms of the associations that automatically influence judgement and behavior when relevant social categories are perceived (Greenwald & Banaji, 1995).
The Stereotype Content Model (SCM; Fiske et al., 2002) suggests that social perceptions and stereotyping form along two dimensions, namely warmth (e.g., trustworthiness, friendliness, kindness) and competence (e.g., capability, assertiveness, intelligence). To determine whether other people are threats or allies, individuals make prompt assessments about their warmth (good vs. ill intentions) and competence (ability vs. inability to act on intentions). The SCM's main tenet is that perceived warmth and competence underlie group stereotypes. Hence, different social groups can be positioned in different locations in this two-dimensional space, since much of the variance in stereotypes of groups is accounted for by these basic social psychological dimensions. For example, in the modern-day US, Christians and heterosexual people are perceived to be high on both warmth and competence, and people tend to express pride and admiration for these social groups. Asian people and rich people are stereotyped to be competent, but not warm.
Elderly and disabled persons, on the other hand, are stereotyped to be warm, but not competent. Finally, homeless people, low-income Black people, and Arabs are stereotyped to be cold and incompetent (Fiske et al., 2007). The warmth and competence dimensions appear to be universal features of social perception and predicted by the structural relations between groups (Fiske et al., 2002).
In three studies, we investigate (1) the general relationship between social stereotypes and hate speech annotation, (2) the particular relationship in curated datasets between social stereotypes and aggregated annotations of trained, expert annotators, and (3) social stereotypes as they manifest in the predictions of hate speech classifiers. We perform these studies with varying measures of stereotypes. In Study 1, we explore whether novice annotators' labeling behaviors are associated with their stereotypes of the target social groups. In Study 2, we turn to language-encoded stereotypes and the aggregated behavior of expert annotators. In Study 3, we then investigate whether the errors of classifiers, having been trained on annotations from experts, are due to the stereotypical bias against targeted social groups. In other words, we investigate whether the biases introduced by social stereotyping, either held by annotators or encoded in language, result in classifiers becoming biased along the corresponding stereotypical dimensions.

Study 1
Here, we investigate the effect of individuals' social stereotypes on their hate speech annotations. Specifically, we aim to determine whether novice annotators' stereotypes of the mentioned social groups lead to higher rate of labeling text as hate speech.
Furthermore, we aim to understand how stereotypes are associated with higher rates of disagreements among the annotators. We conduct a study on a nationally stratified sample (in terms of age, ethnicity, gender, and political orientation) of US adults. First, we ask participants to rate eight US-relevant social groups on different stereotypical traits (e.g., friendliness). Then, participants are presented with social media posts mentioning the social groups and are asked to label the content of each post based on whether it attacks the dignity of that group. We expect the perceived warmth and/or competence of the social groups to be associated with participants' annotation behaviors, namely their rate of labeling text as hate speech and disagreeing with other annotators.

Method
Participants. To achieve a diverse set of answers regarding the eight social groups under study, we selected a relatively large (N = 1,228) set of participants. We recruited participants in a US sample stratified across participants' gender, age, ethnicity, and political ideology through Qualtrics Panels. After filtering participants based on their performance on the quality-check items (described below), our final sample included 857

Stimuli.
To compile a set of stimuli items for this study, we selected posts from the Gab Hate Corpus (GHC; Kennedy, Atari, et al., 2020), which includes 27,665 social-media posts collected from the corpus of Gab.com (Gaffney, 2018), each annotated for their hate speech content by at least three expert annotators. We collected all posts with high disagreement among the GHC's (original) annotators (based on Equation 1 for quantifying item disagreement) which mention at least one social group. We searched for posts mentioning one of the eight most frequently targeted social groups in the GHC: (1) women; (2) immigrants; (3) Muslims; (4) Jews; (5) communists; (6) liberals; (7) African-Americans; and (8) homosexuals. These social groups represent targeted US-related groups in GHC with regard to gender, nationality, religion, ethnicity, ideology, political affiliation, race, and sexual orientation. From the posts that satisfied this constraint, we selected seven posts per group, resulting in a set of 56 items in total. 2 Explicit Stereotype Measure. We assessed participants' warmth and competence stereotypes of the 8 US social groups in our study based on their perceived traits for a typical member of each group. In other words, we asked participants to rate a typical member of each social group (e.g., Muslims) based on their "friendliness", "helpfulness", "violence", and "intelligence". Following previous studies of perceived stereotypes (Cuddy et al., 2007), participants were asked to rate these traits from low (e.g., "unfriendly") to high (e.g., "friendly") using an 8-point semantic differential scale. We  Hate speech Annotation Task. We asked participants to identify hate speech in the 56 items (described in the Stimuli section) after they were given a short definition of hate speech, provided by Kennedy, Atari, et al. (2020): Language that intends to attack the dignity of a group of people, either through an incitement to violence, encouragement of the incitement to violence, or the incitement to hatred.
We verified participants' understanding of the above definition of hate speech by explicitly asking them to verify whether they "understand the provided definition of hate-based rhetoric." Participants could proceed with the study only after they acknowledge understanding the definition. We then tested their understanding of the definition by placing three synthetic "quality-check" items among survey items, two of which included clear and explicit hateful language directly matching our definition and one item that was simply informational (see Supplementary Materials). Overall, 371 out of the original 1,228 participants failed to satisfy these conditions and their responses were removed from the data. The replication of our analyses using the original set of 1,228 participants is provided in the Supplementary Materials. Annotation Disagreement. Throughout this paper, we assess annotation disagreement in different levels: • Item disagreement, d (i) : For an item i, we define item disagreement d (i) as the number of coder pairs that disagree on the item's label, divided by the number of all possible coder pairs. Specifically, if n (i) 1 and n (i) 0 show the number of hate and non-hate labels assigned to document i respectively, item disagreement for document i is calculates as Statistical Analysis. To explore participants' annotation behaviors relative to other participants, we rely on the Rasch model (Rasch, 1993). The Rasch model is a psychometric method that models participants' responses -here, annotations -to items by calculating two sets of parameters, namely the ability of each participant and the difficulty of each item. To provide an estimation of these two sets of parameters, the Rasch model iteratively fine-tunes their values in order to fit the best probability model to participants' responses to items. Here, we use a Rasch model for each subset of items that mention a specific social group, leading to 8 ability scores for each participant.
It should be noted that while Rasch models consider each response as either correct or incorrect and generate an ability score for each participant, we assume no "ground truth" for the hate labels. Therefore, rather than interpreting the participants' score as their ability in predicting the correct answer, we interpret the scores as participants' tendency for predicting higher number of hate labels. Throughout this study we use tendency to refer to the ability parameter, commonly used in Rasch models. Although items' difficulty scores are not assessed in our analyses, we can interpret them as items' controversiality.
We estimate associations between participants' social stereotypes about each social group with their annotation behaviors evaluated on items mentioning that social group, namely, (1) the number of hate labels they detected, (2) their ratio of disagreement with other participants -as quantified by group-level disagreement -and (3) their tendency (via the Rasch model) to detect hate speech relative to others. To analyze annotation behaviors concerning each social group, we considered each pair of participants (N = 857) and social groups (n group = 8) as an observation (n total = 6,856), which includes the social group's perceived warmth and competence based on the participant's answer to the explicit stereotype measure, as well as their annotation behaviors on items that mention that social group. We then fit cross-classified multi-level models to analyze the association of annotation behaviors with social stereotypes. Figure 1 illustrates our methodology in conducting Study 1. All analyses were performed in R (3.6.1), and the eRm (1.0.1) package was used for the Rasch model.

Results
We first investigated the relation between participants' social stereotypes about each social group and the number of hate speech labels they assigned to items mentioning that group. We then analyzed participants' group-level disagreement for items that mention each social group. While the previous analysis uses a Poisson regression model to predict the number of hate labels, here we use a logistic regression model to predict disagreement

Figure 2
The relationship between the stereotypical competence of social groups and (1)  Finally, we compared annotators' relative tendency to assign hate speech labels to items mentioning each social group, calculated by the Rasch models. As mentioned before, by tendency we refer to ability parameter calculated by Rasch model for each participant.
We conducted a cross-classified multi-level linear model to predict participants' tendency as the dependent variable, and each social group's warmth and competence as independent variables. The result shows that participants demonstrate higher tendency (to assign hate speech labels) on items that mention a social group they perceive as highly competent (β = 0.07, SE = 0.013, p<.001). However, warmth scores were not significantly associated with participants' tendency scores (β = 0.02, SE = 0.014, p = 0.080).
In summary, the results of Study 1 demonstrate that when novice annotators perceive a social group as high on competence they (1) assign more hate speech labels to, (2) disagree less with other annotators on, and (3) show higher tendency for identifying hate speech for documents mentioning those groups. Figure 2 represents these associations between annotation behaviors on our stimuli with regard to the stereotypical competence of the social group mentioned in each post. These associations collectively denote that when annotators stereotypically perceive a social group as highly competent, they tend to become more sensitive or alert about hate speech directed toward that group. On the other hand, these annotators tend to be less sensitive about hate speech directed toward stereotypically incompetent groups. These results support the idea that hate speech annotation is affected by annotators' stereotypes against social groups, and specifically the perceived competence of the target social group.

Study 2
Hate speech annotation is marked by exceedingly high levels of inter-annotator disagreement (Ross et al., 2017), which can be attributed to numerous factors, including inconsistent definitions of hate speech, annotators' varying perception of the hateful language, or ambiguities of the text being annotated (Aroyo et al., 2019). As Study 1 suggests, hate speech annotation is affected by annotators' stereotypes toward social groups mentioned in text. Specifically, the results suggest that, for novice annotators, stereotypically perceiving a social group as competent is associated with more sensitivity about and less disagreement on hate speech directed toward that group. Here, we explore the effect of social stereotypes, as encoded in language, on hate speech annotations performed by expert annotators in a large annotated dataset of social media posts.
Contrary to novice annotators, we expect expert annotators to focus more on groups that are the typical target of hate speech; namely, groups perceived as cold and incompetent.
Annotated datasets of hate speech rarely report psychological assessments of their annotators (Bender & Friedman, 2018;Prabhakaran et al., 2021), and even if they do, little variance may exist among the few annotators who code an item. Therefore, rather than relying on annotators' self-reported social stereotypes, here we analyze stereotypes based on the semantic representation of social groups in pre-trained language models, which have been shown to reflect biases from large text corpora (Bender et al., 2021).
We propose that the activation of different social stereotypes upon observing social group tokens leads to higher rates of disagreement. For example, annotators might disagree more when annotating a social media post that attacks "Jews" -a stereotypically cold and competent minority group -whereas they would agree on the label of a post that mentions "Whites" -a stereotypically warm and competent group. We investigate this association using the social groups' stereotypes embedded in pre-trained language models and disagreement in hate labels. We ask, based on Study 1, whether lower tendency for, and higher disagreement on, hate speech directed toward stereotypically incompetent groups is tied to higher inter-annotator disagreement in a large dataset with high rates of hateful language. Figure 3 illustrates the methodology of Study 2, schematically.

Figure 3
The overview of Study 2. We investigate a dataset of social media posts and evaluate the inter-annotator disagreement and majority label for each document in relation to language-encoded stereotypes of mentioned social groups. Contrary to Study 1, stereotypes do not vary at annotator-level.

Figure 5
The effect of mentioned social groups' stereotype content, as measured based on their semantic similarity to the dictionaries of warmth and competence, on annotators' disagreement.

Results
We examined the effects of social group tokens on hate speech annotation based on their quantified social stereotypes. Specifically, we compared post-level annotation disagreements with the mentioned social group's warmth and competence as measured by the cosine similarity of the vector representing the social group token and the vector representing the semantic notions of warmth and competence described earlier. For example, based on this method, "man" is the most semantically similar social group token to the dictionary of competence (Competence(man) = 0.22), while "elder" is the social group token with the closest semantic representation to the dictionary of warmth (W armth(elder) = 0.19). Of note, we investigated the effect of these stereotypes on hate speech annotation in social media posts that mention at least one social group token (N posts = 5535). Since some posts mention more than one social group token, we considered each mentioned social group token as an observation (N sgt = 7550), and conducted a multi-level model, with mentioned social group tokens as the level-1 variable and posts as the level-2 variable. We conducted two logistic regression analysis to assess the impact of (1) the warmth and (2)  In this study, we demonstrated that language-encoded dimensions of stereotypes (i.e., warmth and competence) are associated with annotator disagreement over the document's hate speech label. As in Study 1, annotators agreed more on their judgements about social media posts that mention stereotypically more competent groups. Moreover, we observed higher inter-annotator disagreement on social media posts that mentioned stereotypically cold social groups ( Figure 5). While Study 1 demonstrated novice annotators' higher tendency for detecting hate speech targeting stereotypically competent groups, we found a lower likelihood of hate labels for posts that mention stereotypically competent social groups in this dataset. This discrepancy is potentially due to the fact that while novice annotators are more sensitive about hate speech directed toward stereotypically competent groups (e.g., Whites), these groups are not perceived as targets of hate by expert annotators' majority vote.

Study 3
Previous research has demonstrated that NLP models, trained on human-annotated datasets, are prone to patterns of false predictions associated with specific social group tokens (Blodgett & O'Connor, 2017;Davidson et al., 2019). For example, trained hate speech classifiers may have a higher probability of assigning a hate speech label to a non-hateful post that mentions the word "gay" but are less likely to mislabel posts that mention the word "straight." Such patterns of false predictions are known as prediction bias (Hardt et al., 2016;Dixon et al., 2018), which impact models' performance on input data associated with specific social groups. Previous research has investigated several sources leading to prediction bias, such as disparate representation of specific social groups in the training data and language models, or the choice of research design and machine learning algorithm (Hovy & Prabhumoye, 2021). However, to our knowledge, no study has evaluated prediction bias with regard to annotators' social stereotypes. In Study 3, we investigate whether stereotypes about social groups influence hate speech classifiers' prediction bias toward those groups.
In Study 1, we examined social stereotypes reported by novice annotators to predict sources of variance in annotation behaviors. We discovered less disagreement and more annotator sensitivity about hate speech directed toward competent groups. In Study 2, we investigated the annotated dataset, created by aggregating expert annotators' judgements, and discovered high annotator disagreement and lower sensitivity about hate in documents mentioning social groups which are represented as cold and incompetent in language models. Accordingly, we expect hate speech classifiers, trained on the aggregated annotations, to be affected by such stereotypes, and perform less accurately and in a biased way on social-media posts that mention stereotypically cold and incompetent social groups.
To detect patterns of false predictions for specific social groups (prediction bias), we first train neural network models on different subsets of an expert-annotated corpus of hate speech (GHC; described in Study 1). We then evaluate the frequency of false predictions provided for each social group and their association with the social groups' stereotypes.
During training, BERT's internal parameters and additional parameters, specifically W h ∈ R 768×2 and B h ∈ R 2×1 , which are respectively the weights and bias matrices, are fine-tuned to achieve the best prediction on the GHC. We implemented the classification "White" as the most frequent social group token appears in 16,155 of the predictions and "non-binary" is the least frequent social group token with only 13 observations. We account for this imbalance by adding the log-transform of the number of test samples for each social group token as an offset to the regression analyses conducted in this section.
The average false-positive ratio of social group tokens was 0.58 (SD = 0.24), with a maximum of 1.00 false-positive ratio for several social groups, including "bisexual", and the minimum of 0.03 false-positive ratio for "Buddhist." In other words, models always predicted incorrect hate speech labels for non-hateful social-media posts mentioning 'bisexuals' while rarely making those mistakes for posts mentioning "Buddhists". The average false-negative ratio of social group tokens was 0.12 (SD = 0.11), with a maximum of 0.49 false-negative ratio associated with "homosexual" and the minimum of 0.0 false-negative ratio several social groups including "Latino." In other words, models predicted incorrect non-hateful labels for social-media post mentioning "homosexuals"

Figure 7
Social groups' higher stereotypical competence and warmth is associated with higher false positive predictions in hate speech detection We conducted Poisson models to assess the number of false-positive and false-negative hate speech predictions for social-media posts that mention each social groups. In two Poisson models, false-positive predictions were considered as the dependent variable and social groups' (1) warmth or (2) competence, calculated from a pre-trained language model (see Study 2) were considered as the independent variable along with the log-transform of the number of test samples for each social group token as the offset. The same settings were considered in two other Poisson models to assess false-negative predictions as the dependent variable, and either warmth or competence as the  Therefore, texts that mentions social groups that are perceived as cold and incompetent are more likely to be misclassified as containing hate speech. In other words, a one point increase in the social groups warmth and competence is, respectively, associated with 8.4% and 20.3% decrease in model's false-positive error ratios. Moreover, the number of false-negative predictions is also negatively associated with the social groups' warmth (β = −0.04, SE = 0.01, p<.001) and competence scores (β = −0.10, SE = 0.01, p<.001).
Therefore, texts that mention social groups that are perceived as cold and incompetent are more likely to be misclassified as not containing hate speech; one point increase in the social groups warmth is associated with 3.6% decrease in model's false-positive error ratios and one point increase in competence is associated with 9.8% decrease in model's false-positive error ratio. Figures 7 and 8 respectively depict the associations of the two stereotype dimensions with the proportions of false-positive and false-negative predictions for social groups.
In summary, this study demonstrates that hate speech classifiers trained on annotated datasets predict erroneous labels for documents mentioning specific social groups. Particularly, the results indicate that documents mentioning stereotypically colder and less competent social groups, which lead to higher disagreement among expert annotators based on Study 2, drive higher error rates in hate speech classifiers. This pattern of high false predictions (both false-positives and false-negatives) for social groups stereotyped as cold and incompetent implies that prediction bias in hate speech classifiers is associated with social stereotypes, and resembles human-like biases that we documented in the previous studies.

General Discussion
Social media is one of the most widely-used and powerful channels of communication across the globe, enabling us to express ourselves freely. Although free expression of opinions should be cherished, using online environments to attack, degrade, and call for violence upon other groups is an abuse of this digital public square.
Accordingly, many social-media platforms such as Facebook and Twitter consider "hate speech" to be against their community policies and actively remove hateful content (Margetts et al., 2015). This removal of abusive language online is based on the notion that online hate speech is not an isolated social phenomenon and has been found to be correlated with real-world violence (e.g., Blake et al., 2021). Therefore, there has been an increasing interest in AI technologies, especially in NLP, to develop fair and accurate hate speech detection models by relying on human-annotated corpora and language models to curb online hate (MacAvaney et al., 2019;Ahmed et al., 2021).
Here, we integrate theory-driven and data-driven approaches (Wagner et al., 2021) to investigate human annotators' social stereotypes as a source of bias in hate speech datasets and classifiers. In three studies, we combine social psychological theoretical frameworks and computational linguistic methods to make theory-driven predictions about hate-speech-annotation behavior and empirically test the sources of bias in hate speech classifiers.
In Study 1, we investigated the association between participants' self-reported social stereotypes against 8 different social groups, and their annotation behavior on a small subset of social-media posts about those social groups. We quantified participants' annotation behavior in terms of (a) the frequency and tendency of hate speech labeling, and (b) the extent of their disagreement with other novice annotators. Our findings indicate that for novice annotators judging social groups as competent is associated with a higher tendency toward detecting hate and lower disagreement with other annotators. We reasoned that novice annotators prioritize protecting the groups they perceive as warm and competent. These results can be interpreted based on the Behaviors from Intergroup Affect and Stereotypes framework (BIAS;Cuddy et al., 2007): groups judged as competent elicit passive facilitation (i.e., obligatory association), whereas those judged as lacking competence elicit passive harm (i.e., ignoring). Here, novice annotators might tend to "ignore" social groups judged to be incompetent and not assign "hate speech" labels to inflammatory posts attacking these social groups.
However, Study 1's results may not uncover the pattern of annotation biases in hate speech datasets labeled by expert annotators who are thoroughly trained for this specific task (Patton et al., 2019) and have specific experiences that affect their perception of online hate (Waseem, 2016). In addition, expert annotation of hate speech is a goal-oriented behavior: expert annotators look for cues of derogation and dehumanization carefully to protect minoritized groups. In these cases, annotators actively evaluate not only label correctness but also the consequences of their labeling behavior. In Study 2, we examined the role of social group tokens and language-encoded stereotype content in expert annotators' disagreements in a large dataset containing outgroup-derogatory and dehumanizing language. We found that, similar to Study 1, texts that included groups that are stereotyped to be warm and competent (e.g., Whites) were highly agreed upon by annotators. However, unlike Study 1, we find that posts mentioning groups stereotyped as incompetent -typical targets of hate speech -are more frequently labeled as hate speech.
In simpler words, novice annotators tend to focus on protecting groups they perceive as competent, but expert annotators tend to focus on common targets of hate in the corpus.
To empirically demonstrate the effect of annotation bias on supervised models, in Study 3, we evaluated a hate speech classifier's performance an expert annotated dataset.
We used the count of false-positive and false-negative predictions to operationalize the classifier's unintended bias in assessing hate speech toward specific groups (Hardt et al., 2016). Study 3's findings suggested that stereotype content of a mentioned social group is significantly associated with biased classification of hate speech such that more false-positive and false-negative predictions are generated for documents that mention groups that are stereotyped to be cold and incompetent. These results demonstrate that biased predictions are more frequent for the same social groups that evoked more disagreements between annotators in Study 2. Similar to Davani et al. (2021), these findings specifically challenge supervised learning approaches that only consider the majority vote for training a hate speech classifier and dispose of the annotation biases reflected in inter-annotator disagreements.
Our work has implications for training hate speech classifiers-and more broadly AI and NLP models-because of the legitimate concern that these technologies may perpetuate social stereotypes and societal inequalities (Barocas & Selbst, 2016). Our findings suggest that hate speech classifiers trained on human annotations will also acquire particular social stereotypes toward historically marginalized groups. Our results have three specific and direct implications: First, one way to decrease unintended bias in these algorithms is to diversify expert annotation teams, so that annotators come from different ideological and social backgrounds; by doing so, coders may not agree on a label to (over)protect an ingroup or "ally" groups (leading to false positives), or to ignore actually hateful content targeting "blameworthy" groups (leading to false negatives). Second, supervised learning approaches may benefit from modeling annotation biases, which are reflected in inter-annotator disagreements, rather than the current practice, which is to treat them as unexplained noise in human judgement, to be disposed of through annotation aggregation (e.g., majority voting). The third implication of the present work concerns social science theory development: while societal contexts have been often considered in social science theories, existing theoretical frameworks were not advanced with the vast societal reach of AI in mind. Our work is an example of how existing theories can be applied to explain the novel interactions between algorithms and people. Large amounts of data that are being constantly recorded in ever-changing socio-technical environments call for integrating novel technologies and their associated problems in the process of theory development (Muthukrishna & Henrich, 2019;Bail, 2014;Wagner et al., 2021).

Conclusion
Our work suggests that societal stereotypes are associated with annotators' hate-speech-annotation behavior. Moreover, bias in hate speech detection models is in part due to annotation disagreements rooted in annotators' own biases, namely perceptions of different social groups' warmth and competence. Our framework, combining social psychological and computational linguistic methods, makes novel predictions to understand the effect of human biases on natural language processing models. Our

Study1
Test and Annotation Items.

Muslim
Look forward to reading UR next book. My 1st thought is that we are in this situation through "cock-up" rather than "conspiracy". Or more precisely, "cock-up" on our side & something that's not quite conspiracy on the Muslim side: them just acting as they always do.
We should have learned from History, but we were complacent. Finally, we compared annotators' relative tendency to assign hate speech labels to items mentioning each social group, calculated by the Rasch models. As mentioned before, by tendency we refer to ability parameter calculated by Rasch model for each participant.

Muslim
We conducted a cross-classified multi-level linear model to predict participants' tendency as the dependent variable, and each social group's warmth and competence as independent variables. The result shows that participants demonstrate higher tendency (to assign hate speech labels) on items that mention a social group they perceive as highly competent (β = 0.04, SE = 0.010, p<.001). Warmth scores were only marginally associated with participants' tendency scores (β = 0.02, SE = 0.010, p = 0.098). Except for the significant association of warmth stereotype with the odds of disagreement, other results are the same for both analyses.

Implicit Bias.
To analyze the impact of implicit biases of our pariticipants on their annotations judgments, each participant was assigned to complete an Implicit Association Tests (Greenwald et al., 1998, IAT). Each IAT assesses participants' implicit bias toward one of the 8 groups studied on Study 1 (150 participants for each IAT) The IAT (Greenwald et al., 1998) is a computer-based task designed to measure the strength of automatic associations between two opposing target categories and two opposing attributes.
In each trial, participants are instructed to categorize a stimulus (e.g., a word) as quickly and accurately as possible into one of two target categories or two attributes. In a first combined block, the two target categories and the two attributes are located with a certain associative pattern. In a second combined block, the location of the target categories is switched. A measure of the implicit associations can be obtained computing the difference between the mean latencies of the first and the second combined block. Previous work has shown that IAT scores can be used to assess attitudes and stereotypes, showing adequate levels of criterion validity (Greenwald et al., 2009) and less proneness to impression management concerns compared with self-report measures (Vecchione et al., 2014).
In our study, the IAT provided a mechanism to quantify the participants' bias in preferring a group over a randomly selected pair. The social group pairs assessed in the IAT were the social groups mentioned in the hate speech items (Mexican vs American, Christianity vs Islam, Communism vs Capitalism, Liberal vs Conservative, White vs Black, Gay vs Straight, Female vs Male and Jewish vs Christian). As the result, the calculated IAT score for each participants represents their implicit bias (a value between -1.5 and +1.5) for a specific social group, such that a positive value would represent a higher bias against that group.
We investigated the relation between participants' implicit bias about each social group and the number of hate speech labels they assigned to items mentioning that group.
We conducted a multi-level Poisson model, with social group as the level-1 variables, number of hate speech labels as the dependent variable and the implicit bias score as independent variables. The result shows that a higher number of items mentioning a social group are categorized as hate speech when participants show higher implicit bias for that group (β =0.09, SE = 0.027, p<.01). In other words, one point increase in a participant's implicit bias score for a social group is associated with a 0.1% increase in the number of hate labels they assigned to items mentioning that social group.

Study 2
First, in order to show that annotating hateful rhetoric leads to high levels of disagreement, even among expert annotators, we compared the occurrence of inter-annotator disagreement between hateful and non-hateful social-media posts.
Two-sample permutation tests (5,000 permutations) based on mean and median suggest that annotators disagree more on the posts which are labeled as hate speech (M = 0.50, M d = 0.67, SD = 0.28) compared with those that are labeled as not hateful (M = 0.13, M d = 0.00, SD = .26) by the majority vote (p <.001). This effect is not surprising since hate speech annotation is shown to be a non-trivial task that requires careful consideration of the social dynamics between who generates a piece of text and who perceives the text.
In addition, recognizing non-hate content is substantially easier than flagging inflammatory and potentially hateful content (Waseem & Hovy, 2016).
We then assessed the association of textual mentions of social groups with inter-annotator item disagreements. Two-sample permutation tests (5,000 permutations) based on mean and median suggest that posts which mention social group tokens triggered more disagreement (p <.001) such that, in presence of social group tokens the averaged item disagreement is 0.30 (M d = 0.00, SD = 0.32), as opposed to averaged item disagreement of 0.13 (M d = 0.00, SD = 0.26) for posts without social group tokens.
To examine the interaction of presence of social group tokens and hate speech label, we conducted a two-way ANOVA to compare item disagreements, considering the binary hate label along with the presence of social group tokens as the factors. Both factors (presence of social group token and hate speech content) are significantly associated with item disagreement (p <.001 for both). Figure 9 shows the distribution of disagreement scores for different subsets of the dataset. Generally, mentioning an social group token leads to higher inter-annotator disagreement, however, in hateful posts, mentioning an social group token actually results in annotators agreeing.

Figure 9
Disagreement scores on different subsets of the dataset, based on whether the posts include hate speech and social group tokens (SGT). The horizontal lines demonstrate the error bars.