{'fraction_non_alphanumeric': 0.036861039518900345, 'fraction_numerical': 0.010631443298969073, 'mean_word_length': 4.798412204234122, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
FuzzE: Fuzzy Fairness Evaluation of Offensive Language Classifiers on African-American English

Hate speech and offensive language are rampant on social media. Machine learning has provided a way to moderate foul language at scale. However, much of the current research focuses on overall performance. Models may perform poorly on text written in a minority dialectal language. For instance, a hate speech classifier may produce more false positives on tweets written in African-American Vernacular English (AAVE). To measure these problems, we need text written in both AAVE and Standard American English (SAE). Unfortunately, it is challenging to curate data for all linguistic styles in a timely manner—especially when we are constrained to specific problems, social media platforms, or by limited resources. In this paper, we answer the question, “How can we evaluate the performance of classifiers across minority dialectal languages when they are not present within a particular dataset?” Specifically, we propose an automated fairness fuzzing tool called FuzzE to quantify the fairness of text classifiers applied to AAVE text using a dataset that only contains text written in SAE. Overall, we find that the fairness estimates returned by our technique moderately correlates with the use of real ground-truth AAVE text. Warning: Offensive language is displayed in this manuscript.


Introduction
Offensive language and hate speech pose a significant problem on social media. The use of human moderators does not scale to large online communities (e.g., Twitter). Furthermore, human moderators may write offensive language themselves, thereby corrupting the system. Recent research efforts have focused on annotation theory for offensive language and on developing better classification methods (Davidson et al. 2017;Zampieri et al. 2019). Unfortunately, as companies put offensive language classifiers into production, they may be biased against certain minority groups or linguistic styles (Sap et al. 2019). Yet, fairness is rarely evaluated before putting systems into production for a multitude of reasons. For example, a company or research group may not have the resources to collect data from all demographics of interest, or worse, data for certain groups Copyright c 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. may simply be unavailable or limited for particular topics or social platforms.
Many metrics and strategies have been proposed to evaluate fairness in recent years (Zliobaite 2015;Hardt et al. 2016;Dixon et al. 2018;Mitchell et al. 2019). Most methodologies require ground-truth demographic or linguistic style, annotations (Park, Shin, and Fung 2018;Badjatiya, Gupta, and Varma 2019). In the absence of annotated demographic data, Dixon et al. (2018) propose fuzzing methods to estimate fairness. Fuzzing has traditionally been used in software testing to find bugs or security vulnerabilities (Bird and Munoz 1983). To apply fuzzing to fairness testing, simulated data is used to analyze how predictions change if the topic of the tweet stays the same, but the text is slightly altered. For example, fuzzing techniques will randomly change demographic words (e.g., "He", "She", "husband", or "wife") in a tweet without changing its meaning. If the model's prediction changes by these modifications, then we assume the model is biased.
Typically, fuzzing techniques for software testing use blackbox methodologies (Zalewski 2015;Liu et al. 2019). Yet, the recent fuzzing approaches for fairness relies on manually created templates. Furthermore, current lexiconbased fuzzing methods are limited to single lexical items (e.g., "he" and "she"); complex syntactic constructions are ignored (e.g., "O-be-V"). Likewise, the manual curation process may not capture differences in vocabulary across all minority dialects. For example, "en" in Spanish translates to both "in" and "on" in English. Therefore, Hispanic users may say "Put the soup on the bowl", rather than saying "Put the soup in the bowl". If changing the word "on" to "in" changes the prediction of an advanced classifier, then the model is biased. Without expert domain knowledge, creating fuzzing test cases that capture subtle differences in the use of prepositions may not be obvious compared to words directly related to a specific demographic factor (e.g., "him", "her", "hispanic"). Similarly, differences in the use of language between groups may change over time, further increasing the difficulty of manual curation.
In this work, we investigate the use of style transfer to rank classification models with regard to standardized fairness metrics when minority linguistic styles are miss-ing from the dataset. Intuitively, if a tweet is written in Standard American English (SAE), we want to answer the counterfactual-like question, "What would our model predict if this tweet was written in AAVE?" This task is important because depending on the application, sampling process, and data source, the text generated by specific minority linguistic styles may not be adequately represented in a dataset. Yet, it is important to understand how the model will perform for these groups. Therefore, this line of research can help practitioners ethically adopt machine learning methodologies without dramatically increasing data annotation and collection costs. Essentially, we hope to reduce the burden of evaluating fairness.
Our contributions are summarized below: 1. We present a fairness evaluation framework using fuzzing called FuzzE. Our framework uses style transfer for text. Intuitively, by using style transfer, we can generate AAVE-like text using only SAE data. Our framework can generate a large number of test cases to evaluate how offensive language classifiers will perform on different linguistic dialects. Moreover, we evaluate the use of multiple style transfer methods to estimate fairness as part of our framework. Finally, we provide a simple, yet effective, approach to ensembling multiple style transfer methods to estimate fairness.
2. We conduct a detailed analysis of the framework using automatic style transfer evaluation metrics. Moreover, we measure the increase of well-known phonetic and syntactic AAVE constructions produced by different style transfer techniques after being applied to SAE text. We also perform human evaluation to measure semantic change (e.g., offensive to not-offensive) encountered by transforming the style of text.

Related Work
In this section, we describe three major areas of related work relevant to this paper: style transfer, fairness, and offensive language classification. Style Transfer. Style transfer originates from computer vision, where an image is transformed into a specific artistic style, e.g., an image taken by a cell phone can be made to look like a Van Gogh painting (Gatys, Ecker, and Bethge 2016;Johnson, Alahi, and Fei-Fei 2016). Recently, this idea has been applied to text. For example, many datasets focus on transforming text from a positive sentiment to negative sentiment without changing the underlying topic, or transforming text written from a male's perspective to the perspective of a female. Lample et al. (2019) also transformed text between different age groups, e.g., text written by someone in their 70's is transformed to look as if it was written by a teenager.
In this work, we incorporate state-of-the-art style transfer methods Prabhumoye et al. 2018) into our framework for the purpose of ranking systems with respect to fairness. We note that as new style transfer methods are developed, they can be applied as a drop-in replacement to the methods discussed in this paper.
Fairness. Fairness is an important topic among natural language processing researchers. Bias has been found in word embeddings (Bolukbasi et al. 2016;Zhao et al. 2018;2019), text classification models (Dixon et al. 2018;Park, Shin, and Fung 2018;Badjatiya, Gupta, and Varma 2019), and in machine translation systems (Font and Costa-jussà 2019;Escudé Font 2019). In general, each paper focuses on either testing whether bias exists in various models, or removing bias from classification models for specific applications. However, to measure bias and test bias-removal methods, it is necessary to either annotate or infer the demographic information for each user.
Our work is most similar to Shen et al. (2018), where the authors matched words between two genders, races, and political orientations, then analyze how sentiment predictions change by swapping specific words. In their work, the meaning between two words must be the same. However, there are many words that may appear in SAE tweets, but not AAVE. For example, common proper nouns in SAE tweets may not be discussed in AAVE text. We argue that relying on one-to-one translations between word pairs limits our ability to test the robustness of our models. For example, as previous stated, simple lexical fairness estimation methods do not capture subtle differences in the use of prepositions. Offensive language classification methods should be robust to slight changes in preposition usage as long as offensive tweets stay offensive and vise-versa. Moreover, in this work we focus on fairness ranking whereas Shen et al. (2018) studied the impact of prediction changes across small groups of words.
Hate Speech and Offensive Language. Offensive content is a serious concern for social media companies, government agencies, and online communities. The leading approach to handle offensive language online is to flag such content. Many datasets have been collected and annotated for hate speech and offensive language detection on Twitter (Zampieri et al. 2019;Davidson et al. 2017). Likewise, offensive language and hate speech lexicons have been curated to facilitate offensive language detection (Davidson et al. 2017;Wiegand et al. 2018).
Given the recent interest in classifying offensive language, many methods have been proposed to detect it. Razavi et al. (2010) combined naive Bayes with a multilevel classification strategy to detect offensive language. Gambäck and Sikdar (2017) applied convolution neural networks, showing significant improvements over logistic regression with ngram-based features.
Recently, dos Santos, Melnyk, and Padhi (2018) use style transfer to remove offensive language from the text. This is contrary to work that flags offensive content. While similar to our paper, our work differs in the final application. Specifically, we want to generate AAVE-like language from SAE text. The generated tweets should contain offensive words if they were in the original tweet (i.e., the offensiveness should remain the same).

Datasets
In this section, we provide context on each dataset that we investigate and describe how they are used for training and  evaluating our offensive language detection models with regard to the FuzzE framework. AAVE Dataset (StyleData). Blodgett, Green, and O'Connor (2016) originally collected and released more than 59.2 million tweets by 2.8 million users. Each tweet is accompanied with inferred linguistic style information. Following the work by Elazar and Goldberg (2018), we limit our study to all AAVE and SAE tweets with a confidence of at least 80%. This procedure results in 1.6 million AAVE tweets. We also randomly sample 5 million SAE tweets. The datasets reflect "extreme" differences between SAE and AAVE. We hypothesize that this allows us to test unfair "edge cases" of the offensive language classification models. Moreover, we would expect the offensive language classifier's predictions to be similar for SAE and AAVE tweets that do not differ substantially with regard to style and content; however, this needs to be tested.
StyleData is used to train a Convolutional Neural Network (CNN) Kim (2014) to classify tweets as being written with a SAE or AAVE-like style. Furthermore, this dataset is also used to train the style transfer methods that are used as part of our FuzzE framework.
Offensive Language Datasets. We investigate style transfer and fairness evaluation using two datasets: The Offensive Language Identification Dataset (OLID) (Zampieri et al. 2019) and the Hate Speech and Offensive Language (HSOL) Dataset (Davidson et al. 2017). OLID contains 13,240 tweets labeled using a hierarchical annotation scheme where the top level (task A) differentiates offensive and not-offensive tweets. The bottom level (task C) categorizes insults/threats as targeting an individual, group, or other. For the purposes of this paper, we only utilize the first level, task A, of the hierarchy. The first level contains two classes: "Offensive" and "Not Offensive". HSOL contains 14,509 tweets, each labeled with one of three categories: "Hate Speech", "Offensive Language but not Hate Speech", and "Not Offensive". For the purpose of this paper, and to standardize the outputs across both datasets, we group "Hate Speech" and "Offensive Language but not Hate Speech" into a single "Offensive Language" class.
All AAVE inferred tweets-based on the CNN trained on StyleData-are removed from the OLID and HSOL datasets. The AAVE tweets are used for testing. The SAE tweets in both datasets are split into a training (80%) and test set (20%). We use the data to train/test the offensive language classifiers and rank them with respect to fairness.

Method
The models used in this paper fall into two groups: style transfer and offensive language classification models. The overall workflow of FuzzE is summarized in Figure 1. Intuitively, we propose a tool that takes an offensive language dataset that only contains SAE text, then transforms the SAE text into simulated AAVE (SAAVE) text with the help of style transfer. Both the SAE and SAAVE text are passed to an offensive language classifier to compare the predictions and assess fairness. We briefly describe the style transfer and offensive language models that are part of FuzzE in the following subsections.

Offensive Speech Model
For the offensive language classifier, we train a Logistic Regression (LR) model. Specifically, we train an L2 regularized LR model using tfidf-weighted unigrams and bigrams. Using cross-validation, the regularization parameter is optimized for each dataset independently. We found the best regularization parameters for OLID and HSOL to be 0.1 and 1.0, respectively.

Style Transfer Models
We experiment with four style transfer methods: Backtranslation, Retrieval, Template, and an Ensemble. Each method is trained to transform SAE text to be AAVE-like using the StyleData dataset. This section briefly describes each method we use in our experiments. However, it is important to note that while we present a few methods, as new style transfer techniques are developed, they can be plugged into the FuzzE framework as-is.
Formally, given two datasets X = {x 1 , . . . , x n } and U = {u 1 , . . . , u v } in styles s 1 and s 2 , respectively, we learn a model that transforms X intoÛ = {û 1 , . . . ,û n } in style s 2 . The style-modified dataÛ should preserve the semantic meaning of the sentences in X (i.e., offensive text should stay offensive after processing it).
Back-Translation (Prabhumoye et al. 2018). The intuition behind back-translation for style transfer is to develop a representation of the text that (1.) retains the original meaning of the text and (2.) removes, or reduces, the author's stylistic characteristics from the text. Thus we transform the style from SAE to AAVE using a two step approach. First, following the back-translation framework, we translate each tweet from English into French. We found the translation model used in Prabhumoye et al. (2018) to perform poorly on AAVE text. Therefore, we used Google Translate to transform all 1.6 million AAVE tweets in the StyleData dataset to French. French was chosen to align with the original back-translation model (Prabhumoye et al. 2018).
Second, given the translated tweets, we train a sequenceto-sequence model that learns to translate French into a English text with AAVE-like characteristics. Formally, we learn a model p(u i |z i ) to map between the two styles, s 1 and s 2 , where z i represents the vector representation of i-th french-translated example in style s 1 . The representation is defined as where x f i is the Google Translation of the i-th tweet, θ e is the parameters of the bi-LSTM model, Encoder() represents a bi-direction LSTM model that takes the French Google Translated text and generates a vector z i . It is important to note differences between training and inference. During training, french-translated tweets in s 2 (AAVE) are used as input to the Encoder. However, at test time, french-translated tweets in s 1 (SAE) are used.
Next, given z i , the vector representation of the French translation from Equation 1, we train a bi-directional LSTM decoder Decoder(z; θ d ), where θ d is the parameters of the bi-LSTM model. Furthermore, following Prabhumoye et al. (2018), we use global attention at each step t of the generation processes where h t ∈ R q is the bi-LSTM hidden state for the current time-step t of the Decoder, q is the dimension of the hidden states, andh t ∈ R q represents the Bi-LSTM Encoders hidden state of the source text (i.e., French text). Likewise, score(h t ,h t ) = h T tht represents the similarity between h t andh t . For the model specification of the generator and encoder, we use a twolayer Bi-LSTM with a word embedding size of 300 and hidden dimension size of 500. The generator will create a max sequence of 50 tokens.
Retrieval-based Style Transfer . Retrieval is a TFIDF-based search method that returns the most similar AAVE sentence using a SAE tweet as the query. Specifically, given a sentence x i in style s 1 , we return the most similar sentence u j in style s 2 . Following Li et al. (2018), we only index content words-words that are not indicative of each style. For example, attribute words such as "sholl", "iont", and "sumn" (i.e., words common in AAVE (Blodgett, Green, and O'Connor 2016)) are not indexed. Stopwords are also removed from each query sentence. The retrieved sentence is used as the new stylized version verbatim/as-is.
Template-Based Style Transfer . Template is an extension of Retreival. Using only content words from the input sentence x i of style s 1 , we find the most similar sentence u j in the target style s 2 . Next, the u j sentence's attribute words are used to replace the attribute words in the x i sentence. If the number of attribute words in the retrieved sentence is smaller than the number of attribute words in the query sentence, we use the empty string for subsequent replacements. Refer to Li et al. (2018) for more details.
Ensemble (ENS). Besides the individual models described above, we also evaluate an ensemble method that combines the result of Back-translation, Retrieval, and Template. Specifically, the ensemble is an average of the the fairness metrics we use in Section 5.3: False Positive Equality Difference (FPED) and False Negative Equality Difference (FNED) (Dixon et al. 2018). Intuitively, we are performing model averaging. However, instead of averaging probability outputs, we average the FPED/FNED estimates using synthetic data. FPED and FNED are defined as respectively, where T = {SAAVE, SAE} using the synthetic AAVE text generated by a specific style transfer method. 1 FPR and FNR represent the overall false positive and false negative rates, respectively. F P R t and F N R t represent the group-specific (i.e., SAAVE or SAE) false positive and false negative rates. F P ED and F N ED are calculated independently for each style transfer method by applying the offensive language classifier to each of the style transfer method's generated SAAVE text. The assumption is that the offensive annotations of the original SAE text are still relevant after transforming it into SAAVE (i.e., offensive tweets stay offensive). The ensemble score is calculated as where F XED k represents F P ED or F N ED for the k-th style transfer method and M = {Back-Translation, Retrieval, Template}.
Style Post-Processing. In many cases, after style transfer, the generated text (SAAVE) loses the offensiveness from the original tweet. Thus, we use a lexicon of offensive words and search for them in the SAE tweet before transforming it. 2 For example, using the lexicon, and given the tweet "You are a bitch," we know that the word "bitch" is offensive. The tweet may be transformed into "Y'all be a female" using style transfer. If the offensive tweet becomes "notoffensive", then it is impossible to estimate fairness without human annotation. To ensure that the tweet is still offensive after processing it, we append the offensive words found in the original SAE tweet to the generated synthetic AAVE text. Thus, "Y'all be a female" becomes "Y'all be a female bitch." Post-processing is applied to all of the style transfer methods.

Results
The evaluation strategy focuses on answering three questions: Can state-of-the-art style transfer methods transform SAE tweets into AAVE-like text? If we use style transformed text in place of real AAVE data for evaluation, can we correctly rank the fairest classifiers? Does a better style transform method guarantee better fairness rankings of different models? To answer these questions, we ground our evaluation strategy in the fuzzying evaluation methodology.
Specifically, we use three metrics to evaluate the effectiveness of our framework: • Coverage is generally used to measure how well test cases cover all aspects of a program. For natural language, we quantify how many well-known AAVE characteristics are generated. Moreover, we measure how many AAVE-like tweets are produced. We quantify these coverage quantities in two ways. First, we measure the increase of well-known AAVE phonological variants and syntactic constructions from Blodgett, Green, and O'Connor (2016). Second, we use the CNN trained on StyleData to classify whether a given style transferred string is AAVE-like or not.Via the use of the classifier, we are not constrained to measuring manually curated AAVE linguistic characteristics.
• Pass Rate, in Liu et al. (2019), measures how many generated C programs are valid. For natural language, it is a measurement to see how well meaning is preserved after style transfer. For our experiment, we are not interested in the exact semantics of the original tweet being preserved. For example, if a tweet was originally about Backstreet Boys, but after style transfer it mentions Britney Spears instead, this does necessarily hinder our framework. What does matter is that offensive tweets stay offensive and non-offensive tweets stay not-offensive. Unfortunately, this is not possible to measure automatically. Therefore, we use human annotators to measure whether tweets change between offensive and non-offensive.

OLID HSOL O-be-V gon-V done-V O-be-V gon-V done-V
using synthetic data compared to real AAVE text. We discuss the evaluation methodology for this metric in Section 5.3.

Coverage
In this section, we explore the style transfer method's "coverage" of AAVE characteristics. Three coverage-based metrics are analyzed. First, we measure the increase of wellknown phonetic and syntactic AAVE stylistic characteristics. Next, we apply a classifier that distinguishes between AAVE and SAE text. The classifier is used to automatically measure the increase in AAVE-like characteristics without relying on well-known constructions. Finally, we define a fluency metric, to ensure our models generate realistic text. The results discussed in this subsection (i.e., the numbers in Tables 2 and 3) are calculated using the entire HSOL and OLID datasets (training+test). Phonetic and Syntactic Alignment. Blodgett, Green, and O'Connor (2016) shown that AAVE language on social media exhibits unique characteristics compared to SAE text. For example, AAVE language contains many phonological variants (e.g., sumn, sholl, and iont). We find that the expression of these variants increases after applying the style transfer methods. On the OLID dataset, compared to the original SAE tweets, "sumn" occurs 9 times more often with Retrieval, 9 times more often with the Template method, and 3 times more often with the Back-Translation method. Similarly, in the HSOL dataset, "sumn" appears 20 times more often after applying Retrieval and Template. Back-Translation does not increase the number of occurrences of "sumn" in the HSOL dataset.
We also analyze three well-known AAVE syntactic constructions (Blodgett, Green, and O'Connor 2016): habitual be, future gone, and completive done. We use a Twitterspecific part-of-speech tagger, Twokenizer (Owoputi et al. 2013) to annotate each tweet. In Table 2   dataset, we compare the ratio of each construction in the generated AAVE tweets compared to the original SAE data for the three different style transfer methods. For example, if the O-be/b-V (habitual be) construction appears k times in the original data and h times after processing each tweet using the Back-Translation method, the ratio is defined as h k . All methods produce around 3 times more tweets with the "habitual be" (O-be/b-V) construction. For the "future gone" (gone/gne/gon-V) and "completive done" (done/dne-V) constructions, the Back-Translation method outperforms the other approaches, producing nine times more occurrences in the generated text. The increases on HSOL are not as extreme as seen in the OLID data. Overall, we find that the style transfer methods are able to make the data more AAVE-like based on the in increase of the phonological and syntactic characteristics. Automatic Style Transfer Validation. Translation metrics such as BLEU (Papineni et al. 2002) have commonly been used for style transfer. Unfortunately, we qualitatively found that a better BLEU score does not translate to better style transfer. In some cases, all of the words may change, while the semantic content stays the same. To evaluate the style transfer methods, we train a binary Convolutional Neural Network (CNN) classifier (Kim 2014) using the StyleData dataset that learns to predict whether a tweet is "AAVE" or "SAE". Intuitively, instead of analyzing every possible syntactic variation in AAVE language, we let a classifier implicitly learn the differences between AAVE and SAE. The CNN classifier is trained with 100 filters that span 5 words. If the style transfer methods generate AAVElike tweets, then the CNN should classifier them as such. Furthermore, we evaluate the "fluency" of each style transfer method by analyzing the perplexity based on a pretrained KenLM language model (Heafield 2011) trained on a large external Twitter dataset.
In Table 3, we present the results on the OLID and HSOL datasets. For all three style transfer methods the number of inferred AAVE tweets increases from 3.75% to over 61%-creating 17 times more AAVE tweets than were originally available in the dataset. In terms of fluency, we find that Back-Translation and Retrieval achieve the best performance. This result is expected given Retrieval returns real tweets. Interestingly, the original data has substantially lower perplexity than Retrieval. We believe this is because most of the OLID tweets are in SAE, which is the most prominent language variation on Twitter (Blodgett, Green, and O'Connor 2016). We find similar improvements on the HSOL dataset, with all methods producing more than 60% AAVE-like tweets based on the CNN predictions. Yet, only 21% of the original tweets are classified as AAVE.

Pass Rate
To measure the pass rate, we perform a human study where a single human annotator sampled 100 tweets from each dataset-model combination, for a total of 600 annotations. We sample the same tweets across each model for a given dataset. Moreover, from the 100 tweets we sample, 50 tweets were originally "Offensive" and the other 50 are were originally "Not-Offensive". The human annotator reviewed the tweets after being processed by the style transfer methods. The annotator then relabels each tweet as either "Offensive' or "Not-Offensive" without looking at the original annotations. Intuitively, we want to make sure that the offensiveness of a tweet does not change after applying style transfer. If the offensiveness changes, then the pass rate drops. In Table 5, we show the results of the human evaluation. We find that all methods generally preserve the "offensiveness" of the original tweets. Back-translation performed the best by having an agreement of 0.84 on OLID and 0.91 on HSOL. Here, agreement measures the proportion of relabeled tweets that match the class of the original text before style transfer.

Fairness Estimation
Experimental Setup. It is infeasible to develop hundreds of offensive language detection methods to test on each dataset. Instead, using the training split, we create 100 random samples with replacement (i.e., bootstrap sampling) such that each sample contains 60% of the training dataset. We then train the LR offensive language classifier from Section 4.1 on the bootstrap sampled training splits. Using the SAE test data and the real AAVE tweets, we record the FPED and FNED (See Equation 2 and 3) scores for each model. Furthermore, using the real SAE test data and the SAAVE data-the SAE test data transformed into synthetic AAVE data using style transfer-we also record FPED and FNED scores. Next, we create two ranked lists for each metric, FPED and FNED. One list of ranked scores using the real AAVE data, and the other with SAAVE. We compare the two rankings using two correlation metrics: Pearson r and the Spearman rho correlation. Pearson r is a measure of the linear correlation between two variables. Spearman rho measures the strength and direction of the monotonic relationship between two lists. Results. We repeat the evaluation process 100 times and report the average of each correlation metric in Table 4. Overall, based on the Spearman rho metric, the fairness metric results are positively correlated with real AAVE data. We also visualize the correlations in Figure 2 by analyzing 100 models trained on different subsets of the training data. For both datasets, the ranked FPED/FNED scores are moderately correlated with the ground-truth results. 3 Moreover, on average, Template outperforms both Retrieval and Back-Translation on the OLID dataset. Template also performs comparably to Back-Translation on the HSOL dataset. Overall, we find that averaging the fairness estimates across each method results in the most robust estimate of fairness, at least based on the results in Table 4.

Discussion
In this section, we summarize the results by answering a few questions. First, can state-of-the-art style transfer methods transform SAE tweets into AAVE-like text? In Section 5.1, we show that all three style transfer methods produce a substantial increase in well-known phonological (e.g., sumn) and syntactic characteristics (e.g., O-be/b-V). Moreover, in Table 3, we show an increase in AAVE-like 3 While not directly related, translation metrics such as BLEU generally have correlations in the range of 0.2 to 0.4 (Reiter 2018).

Spearman rho
Spearman rho w/ post-processing w/o post-processing  Table 6: Results on HSOL with (w/) and without (w/o) using a lexicon to constrain each method's outputs.
text. Yet, given the best method only classified 70% of the SAAVE text as being AAVE-like, there is room for improvement If we use style transformed text in place of real AAVE data for evaluation, can we correctly rank the fairest classifiers? In Table 4, we show that the rank of the FPED/FNED scores using SAE and SAAVE text has moderate correlation with the use of SAE and real AAVE text. With only moderate correlation, there is still room for improvement.
Does a better style transform method guarantee better fairness rankings of different models? Again, in Table 4, we find that ENS perform best overall, followed by Template, then Back-Translation. Retrieval performs worse than the other techniques. Overall, we found that the number of syntactic constructions (Table 2) generated by each of the three major techniques was a toss-up, i.e., different models performed better depending on the construction. However, based on the % AAVE results in Table 3, the Template method outperformed both Retrieval and Back-Translation. With regard to pass rate in Table 5, Back-Translation performed best and Retrieval was the worst. Intuitively, to estimate fairness, we find that it is important for successful style transfer methods to generate a large number of AAVElike text based on a classifier as well as keeping the original offensiveness of the original tweet. While Retrieval performed well in Table 3 (i.e., the text as AAVE-like), it performs poorly with regard to pass rate which suggests that  Table 7: Example system outputs using different SOURCE sentences. The outputs include the lexicon-based post-processing trick where we append offensive words there appeared in the source text, but were not in the generated text. a balance of AAVE-likeness and pass rate is important. We hypothesize that better controlled generation that can vary the lexical overlap with the original tweet can help improve the fairness rankings. In the meantime, we suggest the use of an ensemble of style transfer techniques which we find to be the most robust across both datasets.
What is the impact of the lexicon-based postprocessing of the style transferred text? It is important that domain knowledge is incorporated into the style transfer models. In this work, we use an offensive language lexicon. We perform a small ablation study by not adding offensive language words missing in the generated output. The results of the study are presented in Table 6. We find that if we do not add missing offensive words, the overall correlation drops substantially for both FPED and FNED. Furthermore, the FPED scores become negatively correlated with the real rankings. Meaning, based on the fairness measures, that best models are ranked in reverse order. Hate speech and offensive language are generally rare in everyday tweets, at least compared to other topics (e.g., sports). Therefore, style transfer methods are prone to remove offensive words from text. This issue is the motivation behind dos Santos, Melnyk, and Padhi (2018), where the authors show that style transfer is a powerful method of fighting abusive speech, because of their property of removing offensive words. Therefore, for the current state-of-the-art style transfer methods, lexiconbased information should be included for accurate fairness rankings.
Finally, we look at examples generated by the three style transfer methods explored in this work in Table 7. We show cases where style transfer worked (e.g., Back-Translation translates "Bobby Flay" to "Lil niggas" and "you're" to "y'all be") as well as failures (e.g., Template translates "My fucking cousin" to "My her b-day cousin"). Overall, from a qualitative analysis, Back-Translation does a better job of capturing the original semantic meaning of the tweet. Most of the time, the Template and Retrieval methods only capture the same semantic meaning when the retrieved text discusses the same topic. We also have many cases where the offensive word from the original text is removed by the style transfer technique. Therefore, it needs to be added using the post-processing trick. For example, Back-Translation replaces "fucking" with "damn".

Conclusion
Analyzing model behavior across different demographics is important if machine learning is to be used in production systems. This concept is also argued by Mitchell et al. (2019) when creating "model cards". However, if certain demographics are not represented in a dataset, how can we measure fairness? In this paper, we show that style transfer can be used to generate synthetic AAVE text when it is not available in a specific labeled dataset. Overall, while successful to some extent, our main message remains cautionary: if the application can adversely impact minorities, it is vital to manually annotate real-world minority data to measure fairness. The goal of this work is to provide model builders the ability to test a large number of models on a new dataset before investing in human annotation for fine-tuning. Because of resources constraints, many developers may never test the fairness of their models.
There are two major avenues for future work. First, more sophisticated methods should be explored to ensure the generated sentences contain specific offensive words. For example, in this work, any missing offensive word that was available in the original tweet, but removed by a style transfer method, is simply added to the end of the generated text. We could incorporate a finite state acceptor which has been used to constrain neural models for poetry generation (Ghazvininejad et al. 2016). Second, we are interested in analyzing fairness in biomedical areas such as public health surveillance where biased systems can adversely effect policy decisions and people's well-being.