{'fraction_non_alphanumeric': 0.028255742859638396, 'fraction_numerical': 0.010917984103415145, 'mean_word_length': 5.217485745316318, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Examining Stereotypes in News Articles

Gender biases or stereotypes have been studied in short text and manually labeled corpora, but little work has been done in real-world unlabeled text corpora like news articles. This work investigated news articles from mainstream U.S. media outlets ranging from 2013 to early 2020. We used structural topic modeling to estimate gender prevalent topics, compared the results with topic modeling embedding, and incorporated qualitative and quantitative analyses to understand the appearance of gender stereotypes in news articles from each gender group. The structural topic modeling results showed that gender prevalent topics align with stereotypical representations of either gender group and media outlets with imbalanced gender distribution are more influential on stereotypical representations. The topic model- ing embedding results support prior results and provide additional information supporting the conclusion.


Introduction
As machine learning applications are ubiquitous in humans' daily lives, the fairness issues within these models attract many social concerns. While online news articles have become a major source of people obtaining information, the implicit human biases or stereotypes within the natural language could affect the reader's perception of the target. In the domain of fairness issues in artificial intelligence, there is a growing amount of work focused on the detection of human biases or stereotypes with annotated datasets (Dixon et al. 2018;Zhao et al. 2018;Nadeem, Bethke, and Reddy 2020). Many recent works explored the human biases or stereotypes with manually selected corpora but had different definitions of biases or stereotypes depending on the tasks. Moreover, few connect well with related fields like Social Sciences (Blodgett et al. 2020). We follow the proposed social cognitive theory (Bandura, Ross, and Ross 1963) and choose news articles to reflect the real-world setting. According to Bandura, Ross, and Ross (1963), media can be used to discover the human behavioral patterns, which can be treated as a medium of generating and reproducing biases or stereotypes (Foucault 1990). Such biased or stereotypical representations would spread and affect the reader's un-Copyright © 2021by the authors. All rights reserved. derstanding of either gender group through natural language (del Teso-Craviotto 2006).
Hypothesizing that the news articles may contain gender biases or stereotypes, we explored if the author's gender contributes to the stereotypical representation of either gender group. Our work seeks to answer are the gendered topics associated with the stereotypical representations. Instead of identifying the existence of an expected bias or stereotype, we use Structural Topic Modeling (Roberts, Stewart, and Tingley 2019) and author's gender as a covariate, combined with quantitative and qualitative analyses to understand the stereotype appearance by exploring the association between the author's gender and the implicit stereotypes within news articles. We also use topic modeling embedding technique on the same corpora for comparisons. Following Butler (2011), people's understanding of gender categories is constructed through repeated behaviors over time rather than simply the static gender characteristics of women and men, and therefore words associated with women and men contributed to reproducing the ideas of femininity and masculinity even though they do not necessarily reflect realworld settings. Our results show certain topics are more prevalent in either gender group and correspond to those that are considered to be stereotypical representations of females and males (Ellemers 2018). We do not include non-binary gender in this research as we did not find suitable word representations for this group. This work takes a small step forward analyzing biases or stereotypes in a context-dependent setting with topic modeling and extends further by using embeddings on unlabeled textual data.

Related Work
The two general directions of analyzing biases or stereotypes in models are either focusing on the model architecture or the data quality.
Measuring and mitigating biases or stereotypes in embedding space have been tried. Word Embedding Association Test (WEAT) was proposed to measure fairness at the word level (Caliskan, Bryson, and Narayanan 2017). Sentence Encoder Association Test (SEAT) proposed by May et al. (2019) measures fairness at the sentence level. Additional frameworks have been proposed as the extensions of WEAT, for example, contextual word-level embedding association test (Tan and Celis 2019) and Word Embedding Fairness Evaluation (WEFE) (Badilla, Bravo-Marquez, and Pérez 2020). Bias detection and mitigation have been tried in embedding spaces (Bolukbasi et al. 2016;Zhao et al. 2018), but later works indicate not all types of biases are removed from the embeddings (Gonen and Goldberg 2019; Bordia and Bowman 2019;Nissim, van Noord, and van der Goot 2020).
Some works explored the feasibility of topic modeling technique in analyzing biases or stereotypes. Kozlowski et al. (2020) use topic modeling to investigate gender bias in English corpora. Different types of topic modeling frameworks have been tried for further analysis, for example, semi-supervised topic modeling (Andrzejewski and Zhu 2009) has been tried on English and Swedish corpora (Devinney, Björklund, and Björklund 2020) and Davidson and Bhattacharya (2020) examine racial bias and hate speech with structural topic modeling (STM) (Roberts, Stewart, and Tingley 2019).

Methodology
We used structural topic modeling on unlabeled corpora with author gender as the covariate to examine the genderprevalent topics in order to understand if gender contributes to the stereotypical representations. For each media outlet, we estimated an STM to identify the latent topics. We also split each media outlet corpus into two gender subsets and trained two topic modeling embedding using Top2Vec (Angelov 2020). Due to usually accepted stereotypes in the news articles, we hypothesize that certain topics will be gender prevalent.

Data
We used an online dataset containing 2.7 million English news articles from 26 American publications in various domains, including politics, business, and entertainment, ranging from 2013 to early 2020. The dataset contains 221,728 unique author names.
We used the following rules to achieve high confidence in the author's gender identification: • Exclude names that do not list a person's name, for example, "Staff".
These rules exclude both affiliation indicators and genderneutral names, resulting in 140,155 unique names.
In order to identify the author's gender, we used the gender-guesser package in Python which takes first names and predicts possible gender. The package has six output values based on its prediction confidence: unknown, androgynous, male, female, mostly male, and mostly female. For this work, we only accepted author names with "male" or "female" predictions and discarded author names with other output values, such as unknown, androgynous, mostly male, or mostly female. After the gender prediction, we retained 47,713 unique author names with high confidence in the prediction.
Our goal is to discover and understand how the author's gender affects biased or stereotypical content. While preprocessing the documents, we removed unnecessary information Date, Year, Month, Day, Title, and URL, and kept Article content, Author names, and Author gender. All documents were tokenized and each token was converted into lowercase, punctuations and numbers were removed, and the words were stemmed. For the purpose of training STM, we also removed stopwords to improve the performance of the model. We did not remove stopwords while training topic modeling embedding with Top2Vec since they are equidistant from all topics in almost all documents.

Structural Topic Model
While estimating STM, we set the model lower threshold to 5 to filter out words that occur in fewer than 5 documents, as infrequent terms provide less useful information (Davidson and Bhattacharya 2020). In order to mitigate the influence that media outlets with more documents have on the ones with fewer documents, we tested different values from 10 to 30 to find the appropriate number of topics K with STM. We utilized the stm R package to evaluate the results and found that the model converges well with high semantic coherence, low residuals overdispersion, and better prediction performance when K = 20. Therefore, we used the top 20 topics from each model.
We used estimateEffect from the stm package to estimate the gender prevalence of each generated topic within each corpus. We denoted K as the number of topics and M as the number of media outlets. Additionally, we defined K × M matrix A to represent the topic association level in each model by dividing the weights from topic prevalence result into three categories: where µ k,m is the topic prevalence value of topic k within median outlet m. X m denotes the prevalence metadata and γ k denotes the metadata weights. Weak association indicates the number of articles about a certain topic from the group is less or equal to the other group. Moderate association indicates the number is slightly more than the other group. Strong association indicates the number is much more than the other group. We also defined a K × M matrix E to represent the topicgender prevalence in each model: where negative µ k,m value means the topic was male author prevalent and positive µ k,m value means the topic was female author prevalent.

Topic Modeling Embedding
We used Top2Vec (Angelov 2020) to train topic modeling embedding for each media outlet gender group. Top2Vec uses Distributed Bag of Words (DBOW) Doc2Vec (Le and Mikolov 2014) to generate semantic space where the distance between vectors indicates the semantic similarity. The generated semantic space consists of joint word and document vectors to form a continuous representation of topics where the dense areas indicate the topics. We used the same threshold as STM to filter out infrequent terms and the same K value to estimate topics. Unlike estimating with the whole media outlet corpus in STM, we divided each corpus into two subsets based on the author's gender and trained two topic modeling embeddings respectively.

Results and Discussion
We explored the generated topics from STMs and topic modeling embeddings and found that the male author prevalent topics are different from the female author prevalent topics. Additionally, we examined if imbalanced gender distribution affects the stereotypical representations of each media outlet.

Structural Topic Model
Since the corpus contains a large number of news articles from various fields, we expected to see a wide range of topics. Of the 36 generated topics from all media outlets, We were able to identify 20 topics by checking if they align with the news articles content: International Politics; U.S. Politics; Law and Crime; Immigration; Market and Finance; Economy; Education; Technology; Health and Medication; Science; Social Media; Sports; Art; Movie; Music; Literature; Environment; Family and Relationships; Entertainment; and Beauty.
Among all media outlets, we found that topics strongly associated with the male author were grouped in the stereotypical masculine sphere, including International Politics ( state, russian, taliban, iran, nuclear) , U.S. Politics ( presid, republican, democrat, senat, elect) , Law and Crime ( polic, kill, shoot, death, court) , and Sports ( team, game, player, season, playoff ) . We also found that such associations were not exclusive as the female authors also contributed a lot of views in International Politics, U.S. Politics, and Law and Crime. On the other hand, female authors had fewer outputs compared to male authors in Sports, and this showed a stereotypical representation of Sports being masculine (as evidenced by previously reported unequal treatment of male and female athletes).
The female author prevalent topics were mostly grouped within the stereotypical feminine sphere: Music ( song, album, record, band, lyric) , Literature ( literari, poem, book, harri, potter) , Family and Relationships ( mom, dad, home, kid, home) , Entertainment ( show, episod, podcast, disney, comedi) , and Social Media ( youtub, snapchat, instagram, stream, post) . Similarly, we found that Music and Family and Relationships were not exclusive to the female group as some male authors also had outputs about this topic, but such cases were too rare to constitute quantifiable impact.
In order to gain the knowledge of the overall topic prevalence of all media outlets, we combined the topic association level matrix A and the topic-gender prevalence matrix E. The co-occurrence of A k,m and E k,m was shown in Figure  1 with each cell indicating the frequency of certain topicgender prevalence occurs at certain topic association levels among all media outlets. We found that female authors were less influential on political topics. In International Politics, the female group was shown to have more weak associations and less strong and moderate associations, meaning that although many news articles published by female authors mentioned International Politics, the same information was also covered by male authors. As there were more strong and moderate associations between male authors and International Politics, the male group was shown to be more sensitive to international affairs and was able to have more distinct points of view than the female group. Such difference was mitigated in U.S. Politics. Male authors were shown to have less strong associations than female authors, but they had more moderate and weak associations than the female group. This indicates that male authors led the U.S. Politics discussion, but there was a growing number of female authors to publish distinct views in this field that contributed to mitigating the stereotypical representation. Similarly, the difference was also minimized in Law and Crime. Overall, we found that International Politics, U.S. Politics, and Law and Crime aligned with stereotypical masculine representations, and such representations were less significant in U.S. Politics and Law and Crime.
We also identified certain topics that did not align with the expected stereotypical representations. As being closely mentioned with political topics, Immigration was expected to show a similar stereotypical structure. However, the results showed female authors led the discussion as they had more moderate and weak associations than male authors. The topic words of Immigration indicated the topic correlations mitigate the stereotypical representation. Among the news articles that mentioned male author prevalent Immigration topic, the contents were about U.S.-Mexico border control and conflicts, whereas female author prevalent Immigration news articles mostly discussed not only the border control, but also VISA, refugees, immigration regulations, and social impact. The top words extracted from female author prevalent Immigration topics contained terms like family, child, home, provide a potential explanation that Immigration correlated with Family and Relationships, a female author prevalent topic, so that explain female authors had more outputs than male authors on Immigration. In the domain of business, we found that both gender groups had relatively similar views of Market and Finance, but female authors had more outputs about the related topic Economy. News articles content about Market and Finance mentioned more about companies and regulations, whereas the content about Economy concerned more with social events, for example, the employment rate. This showed that Market and Finance correlated with law topics, where the latter was male author prevalent. Female authors, on the other hand, were concerned more about social events, thus had more outputs on Economy than on Market and Finance. Technology was a male author prevalent topic, but the contributions from female authors significantly change the stereotypical representation by providing more distinct views than male authors. Additionally, unlike most male author prevalent technology topics concerned with technical aspects, for example, autonomous vehicles and robots, female authors also discussed the applications of technology in psychology and education. Figure 2 shows a topic prevalence sample from TechCrunch, a media outlet that mostly reports technology-related news. Topics 4,9,10,12,[16][17][18][19][20] were all about Technology, and female author prevalent topics 16-20 were distinct from male author prevalent topics 4, 9, 10, and 12 based on the context. Such a pattern could also be verified by the byproduct of Technology, Social Media, where female authors produced the majority of outputs in this field.
Further analysis examined if imbalanced gender distribution contributes to the stereotypical representations within news articles. We used the topic-gender prevalence matrix E and selected the 5 most stereotypical topics from the prior analysis: International Politics; U.S. Politics; Law and Crime; Entertainment; and Social Media. Results were shown in Table 1. Previous analysis showed that International Politics, U.S. Politics, and Law and Crime were male author prevalent topics, but we found that there were more female authors who talked about International Politics than male authors in 4 out of 11 media outlets, more female authors writing about U.S. Politics than male authors in 7 out Figure 2: Topic prevalence sample from TechCrunch of 17 media outlets and more female authors published articles about Law and Crime than male authors in 5 out of 10 media outlets. Of Social Media, there were more articles from male authors than from female authors in 3 out of 7 media. We hypothesized the imbalanced gender distribution of each media could affect the topic-gender prevalence as imbalanced gender distributions would favor one gender group over the other gender group on topic prevalence.
Additionally, we looked at the gender prevalence of each media to assist further interpretation on topic-gender distribution. Results was shown in Table 2. 85.7% of the media with more female authors discussed U.S. Politics were female author prevalent; 50% of the media outlets with more female authors talked about International Politics were female author prevalent; 60% of the media with more female authors covered Law and Crime were female author prevalent. Similarly, we found all the media outlets having more male authors who wrote about Social Media were male author prevalent. Additionally, 70% of the media outlets that were male author prevalent in U.S. Politics were male author prevalent. In the media that were male author prevalent in International Politics, 50% of them were male author prevalent. In the media that were female author prevalent in Entertainment, 66.7% of them were female author prevalent. In the media that were female author prevalent in Social Media, 75% of them were female author prevalent. This suggested a correlation between topic-gender prevalence and gender distribution in each media outlet: there were 15 out of 20 media outlets showed that male author prevalent distribution would have male prevalence on the selected topics whereas female author prevalent distribution would have female prevalence or equal prevalence on the selected topics. Specifically, in male author prevalent media outlets, most articles about U.S. Politics, International Politics, and Law and Crime were published by male authors. Similarly in female author prevalent media outlets, most articles about Entertainment and Social Media were published by female authors.
Insider Table 1: Topic-gender prevalence on selected topics in each media outlet in structural topic models. "B" stands for Both gender groups. "-" means the topic did not appear. "M" and "F" are male group and female group respectively. 1 represents U.S. Politics. 2 is International Politics. 3 shows Law and Crime. 4 corresponds Entertainment. 5 stands Social Media.

Topic Modeling Embedding
In addition to the 20 topics found in structural topic model results, we were able to agree upon another 2 topics in the topic modeling embedding results: Fashion and Society. Figure 1 contains the results from topic modeling embedding. We found the topics mostly mentioned by male authors were International Politics ( syria, missile, brexit, icbm, korean) , Law and Crime ( gunshot, harassment, suspect, lawsuit, court) , Market and Finance ( ipo, capital, firms, sponsorship, nasdap) , Sports ( championship, nfl, team, score, coach) , and U.S. Politics ( republican, nominee, hillary, bernie, trump) . These findings aligned with previous STM results, indicating a stereotypical masculine sphere consisting of International Politics, Law and Crime, Market and Finance, Sports, Technology, and U.S. Politics.
We also observed the same correlation between topic-  Compared with STM, topic modeling embedding is able to discover more latent topics, for example, Fashion and Society were not identified in STM results. Moreover, it provides more hidden information for analysis by visualizing the semantic embedding. A sample semantic embedding of Business Insider is shown in Figure 3 where a clear boundary divides the female author prevalent topics from male author prevalent topics.

Conclusion
We explored how to use structural topic modeling and topic modeling embedding to understand stereotypes in large unlabeled corpus. With the author's gender as the covariate, we were able to examine how certain topics are disproportionately associated with each gender group. Additionally, we used topic modeling embedding to compare the results. We found that both female and male author prevalent topics aligned with the stereotypical representation of either gender group. We further found that gender imbalance powers the stereotypical representations. The results from topic modeling embedding validated our analysis. Overall, this work demonstrates that unsupervised learning methods can be useful to uncover and assist to understand the implicit stereotypes in unlabeled text corpus.

Limitations
First, we used gender-guesser package to predict the author's gender, kept gender-specific names, and removed genderneutral names based on the results. Although we only ac- cepted the most confident predictions, it is possible that the algorithm could misclassify. Second, it is likely that removing gender-neutral names removes valuable information from the analysis. Finally, we have only examined the gender stereotypes and it is possible that other types of bias exist in the dataset. We hope future works can address these limitations.