{'fraction_non_alphanumeric': 0.03225806451612903, 'fraction_numerical': 0.013541382933188063, 'mean_word_length': 5.149796602057909, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
OK Boomer: Probing the socio-demographic Divide in Echo Chambers

Social media platforms such as Twitter or Reddit have become an integral part in political opinion formation and discussions, accompanied by potential echo chamber forming. In this paper, we examine the relationships between the interaction patterns, the opinion polarity, and the socio-demographic characteristics in discussion communities on Reddit. On a dataset of over 2 million posts coming from over 20k users, we combine network community detection algorithms, reliable stance polarity annotations, and NLP-based socio-demographic estimations, to identify echo chambers and understand their properties at scale. We show that the separability of the interaction communities is more strongly correlated to the relative socio-demographic divide, rather than the stance polarity gap size. We further demonstrate that the socio-demographic classifiers have a strong topical bias and should be used with caution, merely for the relative community difference comparisons within a topic, rather than for any absolute labeling.


Introduction
Social media platforms such as Twitter or Reddit have become an integral part in political opinion formation and discussions, as users exchange opinions on numerous polarising topics such as gun control, abortion or healthcare. This process is accompanied by the forming of echo chambers, i.e. clusters formed by users with a homogeneous content production and diffusion (Cota et al., 2019), where users mostly see posts reinforcing their preexisting belief (DiFranzo and Gloria-Garcia, 2017;Barberá, 2015). At the same time, humans tend to be homophile in their social connections and interactions in general, implying that socio-demographic simi-larities in categories such as age, gender, ethnicity, religion or political ideology significantly increase the chance of a connection between two individuals in social networks (McPherson et al., 2001;Li et al., 2015;Himelboim et al., 2013). So while we do know that similarities foster connections and common identities (Ren et al., 2007;McPherson et al., 2001) as well as that online communities can become echo chambers, by combining network-based community modeling, stance annotations, and socio-demographic projections from natural language of self-identified authors, it is possible to coarsely estimate the extent to which these phenomena inter-play such that this sociodemographic clustering is intensified in online echo chambers. Based on that, we explore the following hypotheses in this paper: (i) key societal topics on Reddit shape network interaction communities indicating the echo chamber phenomenon, (ii) stance polarity mean values are further apart in more separated network communities, (iii) a distinct sociodemographic divide exists between groups of interacting users with diverse stance polarities showing echo chamber characteristics, (iv) automated sociodemographic profiling tools suffer from a strong topical bias, which hinders their ability to characterize the communities.
This paper provides the following contributions: • We create a Reddit dataset of over 20k users (over 2M posts) within 8 current societal topics (Sec. 3), aligned with manual stance polarity annotations of 640 users (Sec. 4). • We quantify the presence and extent of echo chambers in these discussions, employing network-based community detection metrics, such as separability and expansion, and the stance polarity annotations (Sec. 5).
• We develop classification models for sociodemographic variable estimates (age, gender, ideology) and find a strong topical bias, validating their use only for relative comparison of differences between communities rather than absolute labels (Sec. 6). • By applying our socio-demographic classifiers on the detected and quantified network communities, we assess the echo-chamber phenomenon by identifying correlations between the relative difference in socio-demographic variables, the stance polarity differences, and the separability as well as expansion scores of the communities (Sec. 7).

Related Work
Recent works have either studied social-media data with regards to their graph-theoretical properties to detect echo-chamber-like phenomena from the user interactions (Barberá et al., 2015;Colleoni et al., 2014;Conover et al., 2011;Duseja and Jhamtani, 2019;Garimella et al., 2018) or estimated sociodemographic properties of social media users with NLP methods in isolation (Wiegmann et al., 2019;Wood-Doughty et al., 2018;Volkova and Bachrach, 2016;Burger et al., 2011). However, the combination of these two procedures in order to study a potential socio-demographic divide in such user groups at scale has, to the best of our knowledge, not been investigated so far.
In the broader context of analysing the political orientation of users in combination with their demographics, Barberá (2015) studies how Twitter users cluster with respect to different political leanings and shows that women tend to be on average slightly more liberal than men. A similar study demonstrated that there are differences in the average political leaning depending on gender, age, marital status and possession of a college degree (Bond and Messing, 2015), and observed that stronger ties between friends lead to a stronger correlation between their ideologies, which inspired us to the hypotheses explored in this paper.
On a similar note, Bamman et al. (2014) showed that mutual @-connections are more likely to appear between same-gender individuals. Comparable clustering effects were found for age as well as ideology (Li et al., 2015;Himelboim et al., 2013). Furthermore, Bastos et al. (2018) studied the relationship between echo chambers concerning the Brexit referendum on Twitter and the geographic lo-cation of its members, while Ebrahimi et al. (2016) found clear differences in the predicted stance towards Donald Trump between users from different US states, both embodying the idea of extracting social media stance-wise user groups and analysing their characteristics.
Regarding content-based models for the prediction of stance and socio-demographic properties, Durmus and Cardie (2018) studied discriminating tokens in the joined prediction of gender and stance towards abortion, finding that these correlate to the two labels differently, hinting towards our hypothesized topical bias for tokens that correlate more with stance than gender.
In the proposed work, we combine the users' political orientation, their estimated sociodemographic properties and their social media network, while previous works combine only a subset of these concepts, as shown in Table 1. Through a multifaceted analysis of the communities formed, we provide a more spherical insight into the relative differences of their members, in an attempt to analyze political opinion formation.

Dataset Characteristics
We used the API of Reddit to build our dataset. We chose Reddit as our source of data since it provides (i) rich content, due to the fact that there is no word limit, and (ii) a clear relationship between the text and the target topic, since users post within a subreddit. Previous work (Matthes et al., 2018) showed that the controversiality of the topic is one of the main drivers of opinion formation. Therefore, we manually compiled a set of contemporary discussion topics together with subreddits devoted to them (Table 7 in supplementary material for the 8 topics that were included). We crawled threads from these subreddits between November 2019 and June 2021 and periodically extended a database of posts and authors, preserving also the thread hier-  archy.
For the study at hand, we selected the 8 most active topics, and for each of those, we extracted all users with at least 10 posts. The final dataset statistics are provided in Table 2.

Stance Polarity and Intensity Labels
As the opinion of the users towards the investigated topics is a central dimension when studying the echo chamber phenomenon, we discarded our automated stance classification efforts (F1-score around 60% on three classes) and utilized the human labels of the SPINOS dataset instead (Sakketou et al., 2022) 1 . Since the SPINOS dataset resembles a proper subset of the data that will be studied in this paper, it was deemed a feasible source for human labeled user stance samples. The stance labels and their corresponding numeric values are: strongly against (-2), moderately against (-1), stance not inferrable (0), moderately in favor (1) and strongly in favor (2).
The dataset consists of 3526 manually annotated posts from 640 users, which fully overlap with our Reddit data. We analyzed the annotated stances of each user and verified that most users consistently persist on a particular stance polarity. There are a few users with vacillating stances, who seem to mostly persist on one pole (either in favor or against) and express strong stance intensity only for that pole. We therefore compute each user's average stance based on the individual stances of their posts. Figure 1 shows the distribution of the averaged user stances for each topic. We note that, based on 1 https://github.com/caisa-lab/SPINOS-dataset Figure 1: Average user stance distribution per topic the annotation guidelines, a positive stance in the topic of feminism means being in favor of equal rights for all genders, a positive stance against climate change means believing that climate change is caused by humans and constitutes a potential threat on survival and a positive stance in the case of gun control means argumenting in favor of the public availability of guns.

Identifying Echo Chambers
Apart from the stance, a central aspect in the analysis of network structures, and especially echo chambers, in social media datasets is the definition of the interaction itself. Researchers have used retweets (Barberá et al., 2015;Conover et al., 2011) or follows (Colleoni et al., 2014Duseja and Jhamtani, 2019;Garimella et al., 2018) to represent edges between user nodes. These however do not involve an explicit effort of content production (Cota et al., 2019), which is why we, following the work of Trabelsi and Zaiane (2018), focus on replies in the downward subtree of the post to extract the social network topologies. Figure 2 shows an example of how the post-reply tree of a social media post is transformed into the connecting edges of a user interaction network. To choose the optimal amount of communities, we encompass the fluid community detection with a meta-algorithm based on the notion of modularity of the detected communities. The meta algorithm runs the fluid community detection on the extracted network graphs from 2 up until 7 communities, and keeps track of the modularity of the created partitions, which is defined as: where for a certain community c in a graph with m total edges, its number of internal edges is defined by i c , the sum of degrees of the nodes in the community by k c and the resolution parameter γ (Clauset et al., 2004;Hagberg et al., 2008). In the end, the partition with the highest modularity score is returned. We chose modularity since it measures the division of the network into communities, i.e., whether there are only a few connections between the communities, while the nodes within them are densely connected (Clauset et al., 2004), which aligns with our goal of finding echo chambers. To ensure the consistency of our results even despite the elements of randomness in the community detection, 30 runs of the detection function are performed for each community amount, while still maximizing for modularity.
Based on these created partitions, we capture three graph community metrics (Yang and Leskovec, 2015), in order to measure the degree to which each distinct community represents an echo chamber in the network topology on variables that are not explicitly optimized during the community detection. We utilize separability, expansion and density, defined as follows for a community c: here, the number of community-internal edges is defined by i c , outbound edges by o c , and community node count by n c The higher the values for separability and density of a detected community are, the more the interactions of its users are segregated from the rest of the network and rather take place with people from the same "bubble" and therefore represent an echo chamber. Expansion resembles echo chamber effects anti-proportionally as it is increased, when users of a community have more interactions with members from the other groups. To visualize the network nodes, we use the Fruchterman-Reingold force-directed placement (Fruchterman and Reingold, 1991). We further provide the average manually annotated user stance, as described in section 4, for each of the detected interaction communities to explore if these also represent distinct stance clusters. Additionally, a weighted average stance for a community is determined by weighting the sampled stance values by the node degree of the users that contribute them.

Echo Chamber Identification Results
We observe that the Reddit discussions take place in different network topology shapes, not all of them representing the echo chamber phenomenon. Rather, we distinguish three typical shapes: 1. Characteristic properties of the first structure, represented by the topic of nuclear-energy in the studied topics, are low values for separability (≤ 0.5), a high minimum expansion (around 20), a rather uniform distribution of the sampled stances, as well as no visually separated cluster of user-nodes. In discussions of this type, there are no separated communities with opposing stances, rather all discussion participants acting as one community. Such is the case for nuclear energy ( Figure 3a). This can be further validated by the manual stance annotations on this topic, where the majority of the average user stances is 'in favor of the use of nuclear energy' (Figure 1). 2. The second and most frequent structure is characterized by average separability and expansion values, presence of at least one cluster with a rather neutral stance, and visually clearly distinct communities that are spatially close to each other. While for the topics of gun control and Brexit all communities show a similar average sampled stance, in the cases of capitalism and abortion a cluster with strongly parti- 3. The third structure type in the discussion topology, embodied by the topic of feminism in our analyzed data (Figure 3d), resembles the echo chamber phenomenon the most. In this case there is at least one detected community with a high separability score (≥ 4), a low minimum expansion (≤ 2) and at least one cluster with a clearly partisan average stance. The detected communities are also more spatially separated than in the other structure types. Here, likeminded individuals interact segregated from the rest of the network; an echo chamber is formed.

Socio-Demographic Prediction
To study how these community structures relate to their participants' socio-demographic traits we train interpretable supervised classifiers on datasets from previous social-media (Twitter) studies on gender, age, and political ideology (Preotiuc-Pietro et al., 2016;Preoţiuc-Pietro et al., 2017;Preoţiuc-Pietro and Ungar, 2018). For all three of these dimensions, it has been previously shown that social media users cluster along their labels (Bamman et al., 2014;Li et al., 2015;Himelboim et al., 2013). Following previous studies, and considering the available data volumes, we approach these tasks as classification. 2 We acknowledge the suboptimality of predicting binary gender labels and using self-reported training data with users having  only binary option (Larson, 2017). We interpret the predictions in line with (Bem, 1974), examining if the discussion communities differ in constituent features around the class modes. The self-reported labels were obtained through the survey platforms Qualtrics and Amazon Mechanical Turk. Note that the actual posts of the twitter timelines we retrieved for each user might differ from the previous studies.
We predict the user's: (i) self-reported age (three classes: below 30, between 30-45, and 46+), (ii) self-reported gender (male/female), and (iii) political ideology (conservative, moderate, or liberal). Our training data from Twitter for age and gender consists of 3960 users. In order to directly include also users from reddit in the training data, we employ an automatic annotation generation for the dimensions of gender and age group based on regex-matching of 'I am'-statements in the user posts (Welch et al., 2020). For instance we annotate gender by searching for statements such as 'I am a guy/girl' or age with phrases such as 'I am X years old' or 'My grandson/granddaughter'. Users with multiple contradicting 'I am'-statements are excluded from the dataset. This way, we enhance our training data with 966 users from reddit, annotated for gender and 289 users for age. We then enhance the ideology training data with 1223 users from subreddits r/Liberal and r/Conservative, excluding users with less than 5 posts. (2) Unigrams: A user vector is calculated by summing up the appearances of every token, used by at least one percent of the training user base, across all the posts of one user and normalizing these values with the number of posts (Preoţiuc-Pietro and Ungar, 2018).
(3) word2vec: Spectral clustering of word em-beddings creating a feature vector for a given set of posts from a user by calculating the proportion of tokens that belong to each of the topic clusters (Preoţiuc-Pietro and Ungar, 2018). These however didn't outperform the unigram and TF-IDF results.
We intentionally apply only easily interpretable classification models; linear SVM, logistic regression, and random forest. The best-performing setup for each of the user traits, which is used further in paper, is highlighted in Table 4.

Socio-Demographic Prediction Analysis
In the cases of gender and age group, the best-performing predictor(LinSVM) uses unigram-based user vectors, with accuracies of 79% and 58% respectively. For the prediction of political ideology, tf-idf features perform the best with 59%, more than 20% above the random prediction baseline. We then analyze the predictive unigrams, extracting the feature score for each class from the LinSVM coefficient vector as per (Guyon and Elisseeff, 2003;Guyon et al., 2002). The results (Appendix) align with previous work, e.g. selfidentified female users referring more to emotions (Burger et al., 2011;Carpenter et al., 2017).
Furthermore, Table 5 compares the predicted gender distribution of users participating in each topic with the more accurate, but sparser information detected by the regular expressions. We see that our content-based predictor tends to generally over-estimate the percentage of male users for most political topics. The two predictors are in more agreement on the three topics with the lowest amount of male participants, namely abortion, veganism-animalrights and feminism.    Labeling the posts of each user yields a percentage distribution for socio-demographic labels in the communities we extract from the interaction graphs of each topic. Figure 4 shows two examples of the determined socio-demographic distributions of a topic's communities and visually explains how we combined the systems derived in sections 5 and 6 for the following analyses (see Appendix for results on the rest of the topics). Generally, we observe that diverse topics show di-verse socio-demographic community profiles. For Abortion, the violet and green communities have opposing stances and large differences in the predicted gender and ideology distributions. In contrast, for Gun Control, all socio-demographic labels only differ by a small margin.
To formalize our hypothesis that the relative socio-demographic differences between the intratopic community groups grow with the groups becoming more resembling to an echo chamber, we propose to measure, across all 8 topics, the correlation between the separability and expansion values of each community and the average RMSE (Equation 2) of each of the socio-demographic variables (Equation 1) of the detected clusters from the topic's baseline (i.e. the distribution for all users in the topic). A positive correlation in this case means that the more the communities of one topic resemble an echo chamber, the more they also differ in their socio-demographics. In Equation 1, d is the analyzed socio-demographic label, t is a certain topic with the corresponding full user base for this topic b t and the ith detected community c t,i , and pred d (x) is a function yielding the distribution of  Table 6: Pearson's correlation of the separability and expansion values of the detected communities to (a) their mean stance, (b) their stance standard deviation (σ) and (c) their deviations from the full in-topic sociodemographics (Equation 1) label d in a user group: Additionally, we measure the correlation of the two community metrics to the community's difference in stance from the complete topic's contributor average as well as the absolute values of standard deviation (σ). In the four stance-related correlations values, only communities with at least 5 stance-user-samples were considered.

Correlation Results
The results in Table 6 indicate that values of separability and expansion that model the presence of an echo-chamber-resembling interaction network structure (high separability and low expansion) correlate with a larger separation of a sub-community in terms of gender and ideology of the topic's user average. Hence hypothesis (iii) holds -a distinct socio-demographic divide exists between groups of interacting users with diverse stance polarities showing echo chamber characteristics. Furthermore, increased separability and decreased expansion also correlate with a stronger stancewise segregation, confirming our hypothesis (ii) that stance polarity mean values are further apart in more separated network communities. That being said, these communities also show an increased standard deviation of stances, indicating that at least some variance in the opinion of contributing users is present, while more uniform network structures also tend to have more uniform stances.

Discussion and Limitations
The topic-and platform-specific environment underlines the limits of text-based user studies such as ours, indicating a lexical issue in the predictors used, confirming our hypothesis (iv) that the automated socio-demographic profiling tools suffer from a strong topical bias. While words such as problem, understand, or politics tend to be in general statistically more often used by self-identified men (Table 8), this does not hold when comparing discussions within a given topic. Similarly, while words like women, mom or girl are in general strong lexical cues for an author being female (compare Table 8), they tend to be used frequently by both genders just as a part of discussion about abortion or feminism. Similar issues occur with age models, leading to prediction biases. However, note that comparing relative differences (gaps) in estimated demographics between communities within one topic, as we did in Equation 1, is possible, as the bias merely shifts the distribution. In line with (Bem, 1974), we can still examine if the communities differ in constituent features around the class modes.

Summary and Conclusions
We explore the social media phenomenon of echo chambers with regards to its socio-demographic implications. To quantify the forming of these structures, we employed an interaction graph-based algorithm, exploring the separability, density and expansion of the detected communities. For the network topologies of abortion, capitalism, and feminism, we found a moderate to high resemblance of the echo-chamber phenomenon. Bridging the gap between theory and practice, these algorithm and measures could also be used by actual socialmedia platforms to track where its communities are structurally 'echo-chambered' and potential counter-measures are needed. To capture the socio-demographic distributions of the detected communities, we trained interpretable socio-demographic estimation models, scrutinized by keyphrase-based approaches. By merging the network and content information, we found that more 'echo-chambered' topic communities also show an increased separation in their stance and gender and ideology profiles. These results reinforce the call for incorporating socio-demographic and network information into data sets and models for tasks like sentiment analysis, text generation and stance prediction (Hovy, 2015;Hovy and Yang, 2021), while keeping in mind that a lexical topicrelated bias can be a source of misinterpretation in domain-specific user modeling.

A.3 Full results of community detection and socio-demographic prediction
The following are the complete study results for all eight topics. They include the interaction graph with its detected communities as well as a table presenting each communities' user count, weighted and unweighted annotated stance, graph community metrics and predicted socio-demographic distributions. All correlations and analyses in the paper were based on these results.

Cluster
Metrics Sociodemographics  Figure 9: Sampled stance (unweighted and weighted average), separability s(c), density d(c), expansion e(c) and predicted socio-demographics of the detected communities in the discussion around veganism-animalrights on Reddit. The weighted stance is calculated based on the user's degree in the graph