{'fraction_non_alphanumeric': 0.025724725943970767, 'fraction_numerical': 0.007308160779537149, 'mean_word_length': 4.806364922206506, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Behavior Change in Response to Subreddit Bans and External Events

As more people flock to social media to connect with others and form virtual communities, it is important to research how members of these groups interact to understand human behavior on the Web. In response to an increase in hate speech, harassment and other antisocial behaviors, many social media companies have implemented different content and user moderation policies. On Reddit, for example, communities, i.e, subreddits, are occasionally banned for violating these policies. We study the effect of these regulatory actions as well as when a community experiences a significant external event like a political election or a market crash. Overall, we find that most subreddit bans prompt a small, but statistically significant, number of active users to leave the platform; the effect of external events varies with the type of event. We conclude with a discussion on the effectiveness of the bans and wider implications for the online content moderation.


Introduction
Users of online social platforms tend to self-organize into communities. These user-defined communities are a large component of online discourse where users express their ideas and share content about current events. Usually, online communities are organized around a topic or common theme and are governed by rules of behavior and etiquette [15].
We consider the case of Reddit, where online social communities called subreddits commonly sprout up in support of sports teams, movie franchises, video games, political causes, and so on. These communities exist to support and discuss various aspects of the topic with respect to current events. Frequently, external shocks occur which dramatically challenge the identity of the community. Consider, for example, the subreddit in support of the Rams, a professional American NFL football team which moved its home from St. Louis to Los Angeles in the Spring of 2016. When that event occurred, the online community experienced significant upheaval: a new subreddit, r/LosAngelesRams, was created and r/StLouisRams experienced a marked decline in activity. This example raises several socially relevant questions: how many users followed the Rams to the new community? Did existing users disengage from the Rams? and if so, did they join a different fandom? How many Rams-fans left Reddit entirely?
In other cases, there are some subreddits that have been deemed toxic to the larger Reddit community, and have been banned for violating Reddit's community standards. These regulatory actions are known to have a variety of impacts on the respective communities, including changes in posting activity, movement to other communities, or leaving the site altogether.
External and regulatory shocks like these occur frequently in online social communities. Although the story of the NFL's Rams is important to millions of people, the goal of the present work is much broader. We view these events as opportunities to study human behavior and community change as natural experiments. Under perfect conditions, natural experiments like these can uncover causal effects of community and user behavior in response to events. Unfortunately, perfect conditions are rare and confounds are likely present in our data -a fact that we will revisit in the methodology section of this paper.
In the present work, we explore the effect of these external and regulatory shocks on both the individual and community level. To do this, we identified a variety of communities that have been through shocks and track their users to see what effect the shock had on their posting behavior. Specifically we are interested in answering the following research questions: RQ1: How do online communities react to different kinds of shocks? The move of an NFL team (external event) is certainly different from an outright ban of a subreddit (regulatory action). By comparing subreddit bans against external events we can better understand the relative effect of these bans, and we can determine if they are more or less disruptive than the effect of external events.
RQ2: How do individual users react to these shocks? Users of online social systems may react differently to various external events. To answer this question, we analyze which users change their behavior as a result of the ban and attempt to create a model that is capable of predicting whether or not a user will leave the site.
RQ3: How many users change their behavior as a result of these shocks? We use Bayesian Changepoint Analysis to investigate how many users underwent a behavior shift as a result of a shock. These changes could be represented as an increase or decrease in posting behavior, or a change in which subreddits these users are posting in.
Overall, we find that most Reddit bans prompt a small, but statistically significant (one-tailed p < 0.05) number of affected users to leave Reddit (approx. 4% to 10%) while the effect of external events varies with the type of event.
2 Related Work

Digital Traces
Digital traces produced by user activity in online social spaces allow for detailed observational studies of computermediated human behavior. The visualisation of social media analytics is especially important [10]. Location-based social media, for example, has been a helpful way of understanding how events affect various communities [13]. Finegrained studies in community construction [30], inter-group conflict [12], intra-group coordination [21,40], and attention [36,44] have also been made possible by the recent availability of these digital traces of online behavior. Analysis of user behavior has been done on other social media sites such as Pinterest [16], Quora [39], WeChat [24], and Twitter [25]. In one case, the box office success of a film could be predicted using chatter about the movie on social media [25], and information cascades were able to be accurately forecast using data from WeChat [24]. On Reddit in particular, users are have defined to be either content-producers or contentconsumers [37]. Reddit has been used in other social media tasks, such as modelling online comment threads [22] and predicting user interactions [18]. Oftentimes, reactions to events can cause subsequent events in various communities [13].

Reddit Communities
Continuing our focus on Reddit, we note that many subreddits are driven by user retention and growth -by creating an environment where posters and commenters feel welcome and included. To that end, Joyce and Kraut defined three ways by which people continue to post in a community: (1) reinforcement, where a user has positive reactions with others, (2) reciprocity, where a user does something for someone else, and the other returns the favor, and (3) through personal bonds with members [20]. Others have found that stability, cohesiveness, and sociability are important to the future of online communities [5,27]. As communities grow, participation in group discussion tends to become more concentrated [31]. Strong connections to parent subreddits, defined as subreddits that spawn new subreddits, lead to stronger growth in communities [35]. When two communities share users, they tend to be excited about the same things, and interest intrinsically springs up in both groups [2].

Online Toxicity
The freedom of these user-defined communities which provides a space for digital fandoms to flourish also gives way to toxic communities that promote hate speech [5,6,32] and misinformation [34]. Communities address toxic behavior through regulations that constrain activity in different ways [23]. For example, banning subreddits that violate Reddit's anti-harassment policy has shown to effectively reduce the occurrence of hate speech on the platform [6]; however, users of banned communities may simply move to lessregulated parts of the Web [29].
Machine learning models have been created to predict if a subreddit will become hateful or toxic [19]. Likewise, predictive models have been created that can indicate whether or not a conversation is about to turn uncivil [43] or continue to grow in popularity [11]. It is difficult to simply try to find hateful communities by using keyword search, because, counterintuitively, words that appear in hateful communities are also used in supportive communities [32]. Instead, researchers have developed a 'bag of communities' approach which compares hate-filled subreddits to well-moderated subreddits in order to classify other communities as hateful or not [8].

Changepoint Analysis
As an additional part of our study, we look to use changepoint analysis to uncover more behavior change. Besides social media post behavior, changepoint detection has been used in a wide variety of other areas, such as detecting changes in individuals who are at-risk for suicide attempts by analyzing their Twitter posts [38], anomalies in race car driving [41], and sociopolitical events based upon historical documents [9].

Research Framework
We focus our analysis in the present work on Reddit. Users on Reddit self-organize their discussions into topicbased communities called subreddits. By default, posts and comments are sorted by popularity; posts and comments with many (up)votes are viewed by more users, which then have an opportunity to further rate the post or comment [17].
Reddit is an almost ideal use case for our analysis. Information posted to Reddit is publicly available. This data includes the username, the timestamp, and content of the comment, as well as the community in which the comment was posted. Unlike Twitter and Facebook, comments on Reddit are threaded, which provides a simple and intuitive structure from which further information can be gleaned. Additionally, the subdivision and self-categorization of subreddit-communities allows for focused analysis. Community membership and activity evolves over time, rising and falling over time as user interests and perceptions change. Identity-based communities (as opposed to humor or image sharing communities) tend to grow quickly [3] and have high rates of retention [14]. But shocks to this stability sometimes occur. In the present work, we study two types of shocks that occur on Reddit: 1) external events and 2) regulatory actions.
Although the present work focuses on Reddit, the paper may serve as a general framework for the analysis of the effects of external events and regulatory actions on user participation in online social spaces. The events assessed in the present work provide opportunities to perform quasi-experiments to better understand the causal impacts of these externalities on online social systems. Although there are only a handful of popular subreddits that have been banned, there are indeed many subreddits that have experienced external shocks. We only sample a few in the present work. Specifically, our research methodology includes Bayesian changepoint analysis, a difference in differences test which measures changes in user activity, and an analysis of user retention. This framework provides an intuitive look at how social media users continue or change their social media behavior in the wake of a subreddit ban or significant external event.

Quasi Experiments
Much of the work in online social analysis relies on observational methodologies to elicit descriptive statistics. These observational studies are valuable, however many of our questions can only be addressed with causal methods. Unfortunately, randomized controlled experiments of how online users respond to various social stimuli are rare because of the difficulty involved in implementing treatments in massive online social systems.
In an ideal randomized controlled experiment designed to elicit casual effects, a subreddit would have been randomly chosen and either banned it or encountered some topically relevant external shock. Unfortunately, neither of those scenarios is likely to occur, so we use a technique to approximate this ideal scenario. These include 1) the creation of control groups using two kinds of user matching and 2) using a difference-indifferences methods to compute the effect size relative to the control group.

External Events
We define an external event as an event that occurs outside of the digital community that suddenly and dramatically changes the identity of the group. Our goal is to understand how these online communities react to these external events and, later, compare the effect external events with the effect of regulatory actions.
For example, we consider behavioral differences between the r/StLouisRams and r/Chargers subreddits in reaction to each team's relocation. The Rams and the Chargers are two professional American football teams who compete in the National Football League (NFL) who each moved their home to Los Angeles within relatively quick succession, with the Rams moving in 2016 and the Chargers moving in 2017. In this case, the Chargers fans kept their subreddit and Rams fans created a new subreddit: r/LosAngelesRams. Comparing the user retention between these communities is interesting as we compare how these communities deal with the external shock of losing their football team.
Similarly, we analyze how r/StarWars reacted to the opening of the two new films in the "sequel" trilogy including "The Force Awakens" (2015) and "The Last Jedi" (2017). Did Star-Wars community-members change their behavior in response to these external events? And if so, how?
Finally, we analyze how members of r/SandersForPresident and r/The Donald reacted to the 2016 US Democratic and Republican primary elections, in which Sanders did not win and Trump did win, as well as the US general election, which Trump won and Sanders did not participate. Using these examples, we show how politically oriented community members change their behavior after their candidate wins or loses an election.

Regulatory Actions
We define a regulatory action as an action taken by the online social platform to limit or eliminate the behavior of a user or community. Each subreddit is regulated by moderators, who develop community guidelines [7,15] and police their communities for nonconforming behavior [4]. Uneven community standards coupled with Reddit's hands-off regulation had permitted hate speech to flourish in previous years. Additionally, this lack of regulation has allowed for toxic and offensive content to spread through many subreddits [26], which has been shown to depress subreddit growth [28].
In May 2015, following years of controversy about their hands-off approach to regulating speech on their platform, Reddit began to implement an anti-harassment policy. This led to bans of subreddits such as r/fatpeoplehate and r/CoonTown, which had thousands of subscribers and active members at the time of the ban. The r/fatpeoplehate subreddit was banned on June 10, 2015, due to repeated violations of Reddit's hate speech and harassment regulations. After the ban, many users who formerly posted to r/fatpeoplehate created new forums to continue their activity, but these were quickly banned as well; other users left Reddit altogether. Those who stayed used less hate speech in their subsequent activity [6]. However, it is likely that many affected users simply moved elsewhere [6,29,33].
Since this event, Reddit has gone on to ban other troublesome subreddits from its platform. Many of these subreddits are highlighted in Table 1.

Treatment Group
For each subreddit we captured the post and submission history for the year prior and the year after the event. We call these treatment-subreddits. Unless otherwise indicated, we define an active user of a subreddit at the time of the shock as any user who posted at least 10 times in the year prior to the shock event. This default setting approximates the "monthly active user" widely used in Web and social media measurements; however, we find that high-, medium-, and low-activity users do result in different behavior. An active user of a treatmentsubreddit is therefore called a treatment-user.

Control Groups
We compare the behavior of the treatment group against two control groups (A and B). Our goal in identifying these control groups is to find matches to treatment-users so that causal conclusions can be made. Control group A is constructed using the methodology developed by Chandrasenkharan et al [6] which identifies control-users as those who post in similar subreddits with a similar frequency. Following this methodology, we first define a control-subreddit to be all subreddits that a treatment-user posted to in the year prior to the event except the treatmentsubreddit. We then refine this set of control-subreddits to include only those that have at least 10 posts by at least 10 treatment-users, i.e.we only include subreddits where treatment-users were active contributors. We further refine the set of control-subreddits to include only the top 200 with the largest percentage of treatment-users.
A potential control-user is therefore any user who was active (posted at least 10 times in the year prior to the event) in any control-subreddit but who has never posted in the treatment-subreddit. Finally, for each treatment-user, we find a "matching" control-user from the set of potential control-users using the Mahalanobis Distance Matching algorithm according to the logarithm of the user's account age, total karma, and number of posts made in the last year.
Control group B is constructed using the same methodology as control group A, except instead of curating a list of users who post in subreddits similar to the banned subreddit, we only consider those who post at least 10 times in r/AskReddit in the year prior to the event. This restriction identifies control-users who are active users on Reddit and match those that were a part of the banned community but who are less likely to be posting in hateful communities in the case where the treatment group consists of banned subreddits. Again, for each treatment-user, we compute the Mahalanobis distance for each AskReddit user to find a matching set of control-users.

Reaction Measurements
To measure how users react to community shocks like subreddit bans or other external events we plot various measures of user activity starting from year prior to the event until a year after the event. These plots demonstrate the change, if any, rel-ative to the controls and therefore indicate the level of change in user behavior (approximately) caused by the event.
Users may react in various ways to an event. First, the overall activity with the subreddit may change. We can measure activity in the raw number of posts and comments or by the number of unique users making a post or a comment.
When a subreddit is banned, users of that subreddit are (typically) not banned themselves. They may remain on the site but focus their attention on other subreddits instead. Often, alternative subreddits are created in the immediate aftermath of a ban, but these new replacement subreddits are typically quickly shut down so we do not study that behavior in the present work.
We measure the effects of the regulatory and external shocks through a temporal analysis of user posting and commenting behavior. We begin by separating user posting behavior into pre-shock and post-shock periods. We then bin user activity posts into time-windows of 7-day intervals. Our analysis is performed on the mean-averages of user-activity across the bins.

Data
We collected all available posts and comments on Reddit beginning from its inception in December of 2005 through September of 2019 from the Pushshift monthly archives of Reddit data 1 . In total the dataset includes 733,160,370 posts and 6,170,787,222 comments making this one of the largest and most complete online social community datasets available. If a user, comment, post, or subreddit was deleted before the data ingestion began, then that content is not present in our dataset, which is important to remember as we consider the effect on banned communities. Similarly, posts and comments are collected soon after they are created so updates to posts or comments may not be captured.

User Reactions
Here we explore the user-level effects of events and bans (RQ1). We ask: What was the change in number of users? What was the change in overall activity level, i.e., number of post and comments, and type, i.e., where they post? And how does this activity compare between external and regulatory shocks?

Attrition
We first consider the effect that external shocks have on user attrition. That is, in the event of a subreddit ban or external event, how many users leave Reddit altogether? Previous work has shown that subreddit bans have an especially large effect on user attrition [6,29], but these studies only look a few weeks into the future.
Here we define an inactive user as one who ceases to post or comment. Users may continue to visit the site, however, or upvote and downvote posts and comments but that data is not 1 Archived Reddit data collected from https://files.pushshift.io/reddit/ publicly available. In certain circumstances a user may wish to remain for some time and then leave the site, so we repeat our analysis after factoring in a grace-period of 1, 2, 3, and 4 weeks of residual activity before measuring for inactivity. For example, a user of a banned community may complain about the ban in another subreddit for a few days before eventually leaving the site and ceasing all activity. In this case, the short grace period permits this user to be counted as inactive. Figure 1 illustrates the percentage of users who become inactive in the weeks following the ban or external event. The bar plots further break down the attrition analysis according to the number of posts in the year before the shock. We compare the attrition rates of treatment users against the natural attrition of Reddit users in each of the control groups. The attrition rates for many, but not all, shocks were found to be statistically significant using two-sample tests for equality of proportions. Attrition rates that were found to be statistically different from the controls (p < 0.05) are indicated with a left and right † symbol above the bar plots in Fig. 1 representing statistical significance compared to Control A and Control B respectively. p-values are calculated using a two-sided test, but all significant results were from the right side of the distribution; that is, all significant results indicated more attrition compared to the control(s).
Although our previous analysis often found statistically significant user attrition, the overall effect was almost always less than 10%. In other words, at least 90% of users active in subreddits experiencing external shocks or bans continued to post and comment on Reddit. The effect-size was larger for bans on average, but not as much as we expected considering related findings [6]. External shocks. We see that in the case of the NFL teams and the Star Wars subreddits, there was not a significant number of users who left Reddit. However, we do find movement away from Reddit in many of the political subreddits. We can hypothesize that the victory of a candidate caused their supporters to leave Reddit after the election season, because there was no longer a campaign to discuss. We see that in Table 2, there was a statistically significant decrease in users who frequented the StLouisRams subreddit, as compared to the Chargers subreddit. This suggests that even though few individuals left Reddit, there was still a decrease in posting activity, perhaps because the change in team subreddits left many people behind. Regulatory shocks. Most regulatory shocks, or bans, caused a statistically significant number of individuals to leave the platform for most subreddits. One notable exception is the GunsForSale subreddit, which did not show significant attrition. In the case of r/GunsForSale, active gun communities still continued to exist on Reddit, in r/Guns and r/GunAccessoriesForSale (which was created on the day of the r/GunsForSale ban). Perhaps this is why more active users in r/GunsForSale continued to stay on the platform after the ban.
In general we show that the overall effect of a ban was much larger than the effect of external events. We can hypothesize that users of subreddits that experienced an external shock still found Reddit to be a place to go to for discussion, especially since the shocked subreddit continued to exist.

Activity Change
Our next analyses focus on the activity of users that remained active on Reddit. First, we explore the change in the amount of user activity using a difference in differences test. Second, we use multivariate Bayesian changepoint analysis [1] to look for changes in the trends of user activity, where a change is defined as an increase or decrease in posting or commenting behavior or a change in the communities to which the activity is targeted.
The plots in Fig. 2 illustrate the average weekly activity for active users compared to the two control groups. Subreddit bans, shown in the left column, illustrate the overall activity of users comparing the year before the subreddit ban to the year after. Recall that we define active users as those that posted at least 10 times in the pre-ban period. Because no new users can join a banned subreddit we expect to see a natural decrease in the average weekly activity for the treatment group as users leave the platform.
In many cases, however, the difference in pre-ban growth compared to post-ban activity is stark, which indicates that the ban had a chilling effect on the overall posting behavior of the affected users. An important caveat for the analysis of regulatory actions (left column) is that access to a banned subreddit's posts and comments immediately becomes unavailable to our data collection approach in the event of a ban. Although we continue to collect active user activity from other subreddits, we were sometimes unable to collect data for the banned subreddit. This missing data would cause a misleading illustration, so we remove data points for the weeks where we were not able to collect data. For example, in the r/MillionDollarExtreme plot (bottom-left plot in Fig 2) we were not able to collect data for the seven weeks prior to the ban; hence, seven weeks of treatment-group data is not plotted.

Difference in Differences Analysis.
Recall that control groups A and B are used to compare the relative change of the treatment group. Although the results  Fig. 2 are visually compelling, we require a more rigorous statistical test to understand whether these activity changes are statistically significant compared to the controls. In order to account for unmeasured time-invariant confounders we performed a difference in differences (DiD) analysis. The DiD test measures the difference in the effect of a treatment on the treatment group versus the effect on the control groups.
Presumably, the treatment effect on the control groups should be zero, so any effect found in the treatment group should be subtracted from the effect found in the controls. Again consider, as an example, the r/MillionDollarExtreme measurements in the bottom left plot of Fig. 2. Here the large drop in post-ban activity of active users is not matched by a post-ban drop in the controls.
We calculate the mean-average activity in the treatment group (except for weeks with missing data) and control groups pre-ban and post-ban. ∆ pre is the difference between the mean pre-ban treatment and the control activity; ∆ post is the difference between the mean post-ban treatment and the control activity. Finally, DiD is ∆ post − ∆ pre . Therefore, the DiD analysis calculates the effect of the treatment on the post-ban behavior by comparing the average change in the treatment variable compared to the average change in the controls. Table 2 presents the results of the DiD analysis. Negative DiD results indicate that the ban or external event reduced user activity. Conversely, positive DiD indicate that the ban or external event increased user activity.
Our next analysis is to determine if these effects were due to chance. Unfortunately, there do not exist off-the-shelf significance testing for DiD analysis like a t-test. Instead we perform permutation tests to elicit a p-value for each treatment group. To perform these tests, we simulated thousands of treatmentsubreddits from among the potential control-users. Specifically, for each treatment-group we collected the posting histories of all the unmatched active users for each of the 200 subreddits in control group A, i.e., subreddits that were similar to the treatment subreddits. Then we randomly sampled and combined these users to create simulated treatment subreddits. DiD tests were completed between each simulated subreddit to  create a distribution of DiD results. This distribution was then compared to the actual DiD results. The p-value indicates the percentage of the simulated DiD results that were lower than the actual DiD result. Bold DiD results in Table 2 indicate when the DiD effects are significant at the 0.05 level.

Changepoint Analysis
Next, we employ changepoint detection to look for changes that occur in the trend and amount of activity of the treatment groups. Specifically, we use multivariate Bayesian changepoint analysis to identify when users change their online posting behavior [1,42]. Changepoint analysis takes a sequence of observations y 1 , y 2 , . . . , y T corresponding to measurements in a time series then divides these observations into K partitions π 1 , . . . , π K such that the data across partitions is independent: p(y 1:T |π) = K k=1 p(y π k ). Applying these mechanics to our analysis, each observation y i ∈ R d is a d-dimensional vector representing the number of posts and comments made by a user in each of the d subreddits in the i th week before or after the ban or external event date. We use the full covariance model from Adams and MacKay 2 to represent y i as a multivariate Poisson, y i ∼ Pois(λ 0 , λ 1 , . . . , λ d ) [1], where λ x is the mean activity rate for the user in a subreddit. Simply put, the changepoint analysis outputs the probability that a user's posting behavior (i.e., their posting frequency and/or which subreddit they posted to) changed in the i th week. We use a threshold of p(y i ) ≥ 0.90 to draw partition boundaries.
The example in Fig. 3 plots the changepoint of an example user who posted to r/nfl, r/The Donald, and r/TumblrInAction more than 20 times each (as well as dozens of other subreddits that are not illustrated to maintain clarity). We observe that the changepoint probability spikes above the 90% threshold four 2 We reimplemented the original code and make it available at http://github.com/danjr26/changepoint-analysis. Regulatory Actions External Events Figure 5: Probability that a user of a subreddit, posting x-times, experienced a changepoint in the month before or after the shock. Left and right †s over each bar indicate treatment subreddits with changepoint rates that are significantly higher (one-tailed p < 0.05) than the attrition rates of Control A and Control B respectively. For r/greatawakening and r/MillionDollarExtreme, 2 months of data was collected prior to the shock to discover changepoints because of data collection limitations. may yield interesting results. How users are changing, i.e., which subreddits they start or stop posting in, is out of the scope of this work. Simply put, we are currently only interested in if a change occurred, not the nature of the change.
If subreddit bans and important external events significantly affect user behavior, then we should see an increase in the number of users with changepoints identified at or near the ban or event date relative to the controls. Figure 4 illustrates the probability that users in the treatment group and control group A experience a changepoint in the year before and after the ban or event; control group B is not shown in this illustration for clarity, but is very similar to control group A. Note that, like in Fig. 2, Fig. 4 has gaps in some of the banned subreddits because that data was deleted before it could be collected. So, although the significant changepoints in r/greatawakening and r/MillionDollarExtreme are not exactly on the ban date, they are as close as the data allows, and appear to occur approximately when the data for the banned subreddits stopped being collected.
We find that changepoints are relatively rare, with usually less than 2 percent of users undergoing a changepoint in any given week. The upper-bound of 3 standard deviations away from the mean in control group A is drawn as a horizontal dashed line in Fig. 4; we observe that the weeks surrounding the ban or external event do experience a statistically significant uptick in changepoints. While in our previous analysis we saw mixed results on the effect of external events, here we do find that changepoints exists on or near the dates of most shocks. This might correspond to an uptick in activity in these subreddits as users gather to discuss their NFL team moving or the release of the new Star Wars movie. However, we can conclude that there are users who do change their behavior as the result of both external and regulatory shocks.
In some cases, changepoints did not align with our initial expectations. For example, the r/StLouisRams and r/Chargers also had changepoints that correspond with the start of the NFL season, as well as the end of the season when the moves were announced. These changes still correspond to significant external events, just not the events we initially hypothesized. Additionally, we see that many communities have many changepoints outside of the shock. This suggests that users who are posting in these communities might be more volatile in their Reddit posting behavior compared to the control. This may be because these users are joining and leaving new communities more frequently, or varying their posting behavior. More research is needed to uncover why these community members are experiencing changepoints. We also find that the level of engagement in the community increases the likelihood that a user will change their behavior after a shock. Figure 5 illustrates the probability that a user will experience a changepoint in the month before or after the shock, given that they posted between between 10 and 20, 20 and 40, 40 and 80, or over 80 times respectively. In almost all cases, indicated by left and right †, users who are more active in a subreddit are significantly more likely to experience a change compared to Control A and Control B respectively

Predicting Attrition After a Shock
The previous analysis showed that subreddit bans and certain external events have a small but statistically significant effect on user behavior. Our next goal is to determine if there is enough information to predict if a user will stay or leave Reddit after the event (RQ3).

Attrition Results
First, we use standard 5-fold cross validation on the treatment group. In this case, the active users of the treatment subreddit are randomly split into 5 equal-sized sets. In each fold, four sets are combined to create a training set and one set is held out for validation.
We experimented with several different predictive models. Ultimately we chose a straightforward two-layer neural network model, but resisted the temptation to highly optimize hyper-parameters and the model architecture.
From the treatment group users, we identified the m subreddits to which they contributed at least one post or comment. Then, for each user, the input neurons x 1 , x 2 , . . . , x m were set to contain the total number of posts and comments made in each subreddit in the 6 months prior to the ban or external event. The input layer is fully connected to two hidden layers with 1000 neurons each with a RELU activation function. The second hidden layer is fully connected to two output nodes y,ȳ containing a sigmoid activation function. The output layer indicates the likelihood that the user stays on Reddit or does not respectively after a 4-week grace period. We used log mean squared error (LMSE) for the loss function and assigned a prediction label (stay or leave) based upon the output node with the highest weight.
Training is performed for each treatment group indepen-dently under both cross validation and transfer scenarios. Results and 95% confidence intervals for the cross validation results are presented in Table 3. We find that our model can predict user attrition with modest AUC in both the 5-fold cross validation and transfer scenarios. This suggests that it is possible to predict which users will stay leave the platform, given past posting history, with fair accuracy.

Conclusions
We conclude by revisiting the research questions asked at the beginning of this paper. RQ1: How do online communities react to different kinds of shocks? Although previous work has shown that subreddit bans are very effective at removing hateful content from Reddit [6], we find that subreddit bans and external events typically result in only a small decrease in the number of users and number of posts per user. Combined, these results suggest that subreddit bans may be able to refocus users' attention towards producing non-hateful content rather than simply removing them from the site. For those that do make a change, it remains unclear how they change.
We also observe several interesting changes in activity related to external events. For example, members of the r/SandersForPresident subreddit have a larger decrease in activity compared to the r/The Donald, due to Sanders' failed candidacy and Trump's nomination. In the r/StLouisRams subreddit have far less activity after the move to Los Angeles than the r/Chargers subreddit. It is likely that, by keeping the r/Chargers subreddit name and identity, there existed a coherent space for users to continue discussing football and their teams' move. Additionally, we see that before the teams announced their move, active members of r/Chargers and r/StLouisRams community both had changepoints that correspond to the start of the NFL season.
The Bitcoin crash also had a large external effect on r/DarkNetMarkets, as illustrated in Figure 1. This result suggests that the Bitcoin crash did have a significant impact on the amount of time that those users spent discussing Bitcoin. After r/DarkNetMarkets was banned, just a few months later, active users decreased their overall posting behavior even further. RQ2: How do individual users react to these shocks? Despite these statistically significant findings, we were surprised to observe that most users continued their usual posting behavior. Changepoint analysis found that most users did not change their behavior. The largest effect occurred during the ban of r/greatawakening, in which 10.2% of affected users had a changepoint registered. In most communities, less than 5 percent of users experienced a changepoint as a result of the shock. RQ3: How many users change their behavior as a result of these shocks? We observed that a small, but statistically significant, number of active users will leave Reddit after their subreddit is banned. Unsurprisingly, these effects are higher if the user contributed more frequently to the banned subreddit. However, the large majority (typically 90% or more) of active users who were active in these banned subreddits do stay on the platform -even after accounting for a generous grace period. These results suggest that even though Reddit bans do rid the platform of many of the dedicated members of these offensive subreddits, as seen in Figure 1, most stay around despite the ban. We also observe, in most cases, a small, but statistically significant, reduction in the number of posts and comments made by those active users of a banned subreddit.
It is important to note that the present work is limited by its selection of subreddits and events. Although there are only a handful of popular subreddits that have been banned, there are indeed many subreddits that have experienced external shocks. We only sample a few in the present work. However, the methods used in this paper constitute an expanded framework that can be used to study online communities in future work.
It is also important to note that our analysis only captures the amount and frequency of posts and comments that are made on Reddit. We do not consider the words used in each activity, nor do we observe the subreddits that people are browsing, or the posts and comments they upvote or downvote. As is typical in Web culture, most social media users are passive viewers of content [17], so we do not claim to capture all user activity. However, by using post and comment activity history, the present work can understand the effect that external events and subreddit bans have on user activity.