{'fraction_non_alphanumeric': 0.02322409275279084, 'fraction_numerical': 0.007071323450231523, 'mean_word_length': 5.113671274961598, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Beyond Keywords : Tracking the evolution of conversational clusters in social media

The potential of social media to give insight into the dynamic evolution of public conversations, and into their reactive and constitutive role in political activities, has to date been underdeveloped. While topic modeling can give static insight into the structure of a conversation, and keyword volume tracking can show how engagement with a specific idea varies over time, there is need for a method of analysis able to understand how conversations about societal values evolve and react to events in the world, incorporating new ideas and relating them to existing themes. In this paper, we propose a method for analyzing social media messages that formalizes the structure of public conversations, and allows the sociologist to study the evolution of public discourse in a rigorous, replicable, and data-driven fashion. This approach may be useful to those studying the social construction of meaning, the origins of factionalism and internecine conflict, or boundary-setting and group-identification exercises; and has potential implications for those working to promote understanding and intergroup reconciliation.


Motivation
Social media suggests a tantalizing prospect for researchers seeking to understand how newsworthy events influence public and political conversations.
Messages on platforms such as Twitter and Facebook represent a high volume sample of the national conversation in near real time, and with the right handling 1 can give insight into events such as elections, as demonstrated by [Huberty, 2013, Tumasjan et al., 2010; or processes such as social movements, as demonstrated by [Agarwal et al., 2014,DiGrazia, 2015. Standard methods of social media analysis use keyword tracking and sentiment analysis to attempt to understand the range of perceptions surrounding these events. While helpful for basic situational awareness, these methods do not help us understand how a set of narratives compete to interpret and frame an issue for action.
For example, if we are interested in understanding the national conversation in reaction to the shooting at Emanuel AME church in Charleston, South Carolina on June 17, 2015, we could plot a timeseries of the volume of tweets containing the hashtag #charleston, as seen in Figure 1. This tells us that engagement with the topic spiked on the day of the news, and then persisted for several days before returning to baseline. It does not tell us that the initial spike may have been retellings of the details of the event itself, with the tail comprised of interpretations and framings of the event in the political discourse, the ways the event is interpreted by various groups, or how these groups connect the event to their preexisting ideas about gun violence. Additionally, it does not let us track continuing engagement with the follow-on ideas stimulated by the original discussion. Figure 1: Standard methods of social media analysis include keyword volume tracking 3 (shown here), sentiment analysis, and supervised categorization Alternate methods include assessing the relative 'positive' or 'negative' sentiment present in these tweets, or using a supervised learning categorizer to group messages according to preconceived ideas about their contents, as demonstrated by [Becker et al., 2011,Ritter et al., 2010,Zubiaga et al., 2011.
Such methods can give aggregated insight into the sentiment expressed, but fail to uncover from the data the ways the events are being framed within existing conversations.
Because they look at counts of individual messages and not at the relationships between ideas, techniques which focus on volume, sentiment, or category are unable to uncover coherent patterns of thought that signify a society's interpretation of the events' deeper meanings. To interpret events, individuals must make connections between an event and historical parallels or concepts in the public discourse. It is thus the expressed connections between ideas, not merely the ideas themselves, which must be tracked, categorized, and interpreted as samples from an underlying semantic structure.
Our approach inverts the investigation of structures of connected persons observed in social networks, as demonstrated by [Zachary, 1977] or [Morales et al., 2015], and instead investigates structures of connected ideas within semantic networks. 4 While the individual-level semantic networks which constitute personal interpretation of events are fundamentally unobservable, messages sent by individuals encode samples of these connections in an observable format. These individual-level samples can be aggregated to form a societal-level network representing the superposition of active semantic network connections of the society's members. For the purpose of this paper, it is sufficient to note that if structure is discernible in this aggregate then structure is implied within the members, allowing that no one individual need represent more than a subset of the aggregate structure. 5 4 For a theoretical discussion of semantic networks as representations of human knowledge, see [Woods, 1975], [Mayer, 1995] and [Schilling, 2005], for a physiological description see [Tulving, 1972] and [Collins and Quillian, 1969]. 5 The fact that macroscopic structures do exist in the aggregate representation is itself 4 One way to aggregate these connections is to look at a network of word co-occurrences in social media messages, as has been demonstrated by [Cogan andAndrews, 2012, Smith andRainie, 2014]. This technique represents references to an event, to analogous historical events, and to other public discourse concepts as nodes in a semantic network. Each message containing two concepts contributes to the weight of an edge between these nodes. The structure that results gives a macro-level aggregated sample of the microlevel structures of meaning within individuals' own minds. For example, Figure 2 shows the network of hashtags formed around the focal concept of the Charleston shooting, on June 18, 2015. Each hashtag in the conversation is represented as a point or 'node' in this diagram, and each message containing a pair of hashtags contributes to the strength of the link or 'edge' connecting those two nodes. 6 Within this example appear two distinct clusters of connected ideas. In this case, the upper cluster represents a description of the shooting itself and the human elements of the tragedy, and the lower cluster focusses on the larger national-scale political conflicts to which the event relates. Within each cluster, ideas relate to one another, and connections give context to and comment upon one another. We might consider this the essence of surprising. One could imagine that such an aggregate would resemble a random network, and the fact that this is not the case suggests an underlying sociological process for the social construction of meaning. A full description of how these processes may operate is forthcoming by the first author. 6 Here we use a cutoff threshold to convert a frequency-weighted set of connections into an unweighted network diagram. Figure 2: Coherent structures in the aggregate hashtag co-occurrence network can serve as proxies for 'conversations' in the larger societal discourse, and give insight into the structures of meaning within the minds of individual members of that society what is meant by a national conversation around a topic -a set of mutually acknowledged statements of meaning. In contrast, across clusters we see fewer connections between ideas, suggesting that the statements made within one cluster do not inform or relate to those from the other. By examining the structure of the semantic network, we begin to see that instead of one national conversation reacting to the event, there are (at least) two conversations going on, each strongly connected to itself, but only weakly relating to the other. 6

The contributions of this paper
We may hypothesize that how these conversations evolve and interact will influence the social and political response to the event they describe. In this paper we build on the qualitative example above, and present a formal method for identifying conversational clusters, describing their structures, and tracking their development over time. We present various methods for visualizing conversational structure to give insight into how certain elements form the central themes of a conversation while other elements circulate on the margins. We show how overlapping conversational clusters may be identified, and explore their relevance to discussions of factionalism and consensus building. We then demonstrate a method for tracking the evolution of these conversational clusters over time, in which clusters identified on one day are compared with clusters emerging on subsequent days. This allows us to identify how new concepts are being included into the discussion, and quantitatively track engagement in full conversations as distinct from mere references to keywords.
Formally representing elements of a conversation as being frequent vs. rare, or central vs. marginal, and tracking how these metrics develop over time, allows the sociologist to study the way new ideas obtain relevance. For example, a central and well-connected topic may become prominent through a process of amplification, or a related popular topic may become relevant through a process of linking. These phenomena, which happen societally at the level of individual terms, can then be studied as drivers of macro-level phenomena such as factionalism, polarization, and realignment. By tying the changes in conversation to changes in underlying social structures, the narrative identity-forming and boundary-setting activities of groups may be studied.
In the appendices to this paper we provide example scripts that save those wishing to use these techniques from the burden of reimplementing these algorithms. Due to the computation-intensive nature of this analysis, we chose to implement the data manipulation algorithms in both Python and Unicage shell scripts, 7 for prototyping and speed of execution, respectively.
Descriptions of these scripts can be found in the appendices, along with performance comparisons between the two languages.

Identifying Conversation Clusters
The network in Figure 2 represents connections individuals have made between hashtags. 8 Each of the hashtags present in the dataset forms a node in this network, and the relative strength of edges depends upon the number of times the pair occur together in a tweet, their 'co-occurrence', using the method of [Marres and Gerlitz, 2014]. 9 To formally identify the clus-7 For a description of Unicage development tools, see [Tounaka, 2013] 8 The attributes of these connections are not considered in this analysis, and may thus be positive or negative, binding or exclusive. 9 It is of course possible to conduct the analysis using the full set of words present in a tweet, omitting stop-words, or focussing purely upon easily identifiable concepts. From an analytical perspective, this has the effect of expanding the scale of the clusters, as a broader range of concepts are included. Within this paper, we limit the analysis to hashtags purely for the purposes of simplifying the presentation, and keeping visualizations to an 8 ters visually present in the diagram, we apply k-clique community detection algorithms, implemented in the COS Parallel library developed by [Gregori et al., 2013] and demonstrated in the appendices.
Clique percolation methods work to identify communities of well-connected nodes within a network in a way that allows for the possibility of overlapping or nested communities. The methods allow the analyst to create metrics that identify communities based upon local network characteristics and are thus invariant to network size or connectivity outside of the local community. As the measured extent of a conversation should be determined by the content and connectivity of that conversation, and not by the structure of unrelated discussion, this method of community detection is for our purposes superior to those based upon graph cutting or intersubjective distance measures. 10 The K-clique percolation method suggests that communities are composed of an interlocking set of cliques of size k, that is k nodes which are all mutually connected to each other; and that the overlap of each clique with a neighboring clique is (k-1). This ensures that each member of a community is connected to at least (k-1) other mutually-connected nodes within the community, and that any sub-communities are connected by a bridge of at appropriate size. We are grateful to an anonymous reviewer for highlighting that a full text analysis not only allows for a larger sample of messages to be observed within the conversation, but is essential for researchers studying how concepts rise to prominence, as the initial process of concept formation or linkage is likely to occur before the concepts are institutionalized with a hashtag. A comparison of the full-text and hashtag-only analysis is present in Appendix F.
10 For a comparison of community detection algorithms and their features, see [Porter et al., 2009]. 9 least width k.
The metric k determines how strict the conditions are for membership in a cluster, and thus drives cluster extent and boundaries. For example, high values of k would impose strict requirements for interconnectedness between elements of an identified conversation, leading to a smaller, more coherent identified conversation, as seen in Figure 3. Each member of this cluster is connected to at least 4 other members who are all connected to one another. On the other hand, smaller values of k are less stringent about the requirements of connectivity they put on the elements in the cluster, leading to a larger, more loosely coupled group as seen in Figure 4, whose members must be connected to at least two other mutually connected members of the community.
When we apply these methods to word co-occurrence networks, the result Figure 4: Clusters with lower k value are larger and less less tightly connected, representing more diffuse conversation. They may have smaller clusters of conversation within them is a collection of sets of words which all form a conversational cluster. As the extent and connectivity of the identified cluster is dependent upon the value of k used to create it, by varying k we can identify closely connected subsets that represent the central structures of the conversation, and occasionally multiple subsets representing internal factions within the discussion.
As the algorithms depend only on the presence, not the strength of connections within the conversation, information of the volume of messages making a semantic connection can be encoded as a threshold weight w for inclusion into the analyzed co-occurrence network. By varying w we can identify the extent to which a connection is recognized by members of the population. To demonstrate the utility of these parameters we explore their extrema. Clusters that form with high values of k and low values of w may be considered central to the conversation, but only by a minority of the population. Conversely, concepts contained only within clusters with high values of w and low values of k are universally agreed to be marginally related to the focal discussion.

Representing Conversational Clusters as Nested Sets
Tight conversational clusters (high k) must necessarily be contained within larger clusters with less stringent connection requirements (lower k). Performing clustering along a range of k values allows us to place a specific conversation in context of the larger discourse. It becomes helpful to represent these clusters as nested sets as seen in Figure 5, ignoring the node and edge construction of the network diagram in order to display the nested and interlocking relationships the conversations have with one another. In this diagram, members of a 5-clique cluster are arranged (in no particular order) within the darker blue box. These form a subset of a larger 4-clique cluster which also includes a number of other concepts.
With this representation we see elements of the conversation that are more well-connected and thus 'central' to the conversation, along with those Figure 5: Converting networks to nested sets based upon k-clique clustering simplifies presentation and analysis of various levels of conversation relating to one another less tightly. Existing methods which look at volume may suggest that ideas are central to a conversation if they are well represented in the sample. These methods lack the ability to analytically discern that different topics are located within the core of different conversations.
Within the nested structure presented here, central topics are defined not by their volume, but by the multiplexity of their connection with other concepts which also form the core of the discussion, elements to which they relate and whose meaning they are seen as essential to. Within the larger conversation, peripheral terms are less mutually interdependent, and less essential for understanding.
Specifically, this method allows the qualitative observation of conversational clusters that we drew from Figure 2 to be formalized into an explicit and analytically tractable structure. The upper cluster of our example is highlighted here, and we show that this cluster is actually composed of a more tightly connected set of central ideas referencing the event itself and the immediate response it elicits, embedded within a broader set of ideas additionally referencing the perpetrator and victims. 11 In noting that the abstractions of terrorism and racism are part of the center of this part of the conversation, with the specific details closer to the margins, the sociologist may wonder if this is because the event is seen primarily as an instance of a more important phenomena than as an interesting occurrence in its own right.

Tracking Conversations Chronologically
In order to track how elements of conversation weave into and out of the general discourse over time, we need to be able to interpret how conversational clusters identified at one point in time relate to those in subsequent intervals. We can do this in one of two ways.
The first method is to track the volume of co-occurrences identified in the various conversational clusters identified on the first day of the analysis, as it changes over subsequent days. This indicates how well the connections made on the first day maintain their relevance in the larger conversation. Figure 6 shows how the connections made in conversational clusters on June 18th fall in volume over the 10 days subsequent to the initial event, paralleling the decay in pure keyword volume seen in Figure 1.  We should thus interpret the transition matrix not as a way to select the single subsequent cluster out of many which represents the original cluster These dynamics point to underlying processes of polarization, factionalism, or reconciliation within the population engaging in the discussion. We should expect that in the process of meaning-making with regards to an event, the emergence of consensus should be characterized as a move toward fewer clusters as some clusters merge and others fall out of the larger discussion. Increasing polarization suggests the fission of conversations into partially overlapping clusters, as individuals from each faction begin to talk past one another, highlighting different aspects of the overarching conversation.
We can extend nested cluster diagram shown in Figure 5 to illustrate how On June 22, the shooting itself has become less tightly linked with political discussion, as it is replaced with prognostic framings and calls to solidarity actions ('charlestonstrong', 'unitychaincharleston', 'bridgetopeace'). In response to political pressure, the Confederate flag is removed from the state capitol. On June 23rd, the tight clustering of prognostic framing linked with the references to the shooting has dissipated, and the conversation returns to its longer-term structure, with movement references again moving toward the margins of the discussion.
Having explored the way the conversation that was formed in response to the shooting evolved into a conversation about the state's support for symbols of racism, we can finally return to the original task of assessing 21 public engagement with the discussion. In figure 9 we show the volume associated with each of the sub-conversations and some of the key terms associated with their evolution. Here we observe that in shifting its emphasis from a conversation about the event alone to a discussion with more political relevance, overall engagement with the topic actually surpasses the original discussion about the event on the first day.
This leads to the sociological conclusion that rather than the typical news cycle, what we see in this case is actually the trigger of a small wave of social activism linked to a larger social movement. that gained steam following the Charleston shooting was not a single discussion of a single set of issues. By structuring as a semantic network the way various concepts built and reflected on one another, we recognized that there may have been be multiple simultaneous but separate conversations addressing different aspects of the tragedy. By examining the density of relationships within each conversation, we found the central aspects of each conversation. By tracking the evolution of these conversations over time, we saw how events became linked to larger social issues, and how these links drew on prior discussion to frame and suggest responses to the events. These conclusions can only be drawn by understanding the links between the event and its interpretations as they evolve over time.
The utility of social media analysis for sociological research can be extended well beyond the practice of tracking keyword volume, net post sentiment, or supervised classification. In particular, tracking conversational clusters within a network of idea co-occurrences can give both structural understanding of a conversation, and insight into how it develops over time.
These tools can prove helpful to sociologists interested in using social media to understand how world events are framed within the context of existing conversations.
Follow-on work to this paper could attempt to separate the various con-

Author's Note
The appendices to this paper contain all of the code needed to collect necessary data, generate the analysis, and and produce visualizations found herein:

Appendix B Cluster Identification and Transition Analysis in Unicage
Appendix C Performance Comparison Between Python and Unicage Examples

Appendix E Visualizations
Appendix F Comparison of hashtags-only analysis with full analysis The full set of scripts, and associated documentation, can be found at https://github.com/JamesPHoughton/twitter-cluster. This code takes messages that are on twitter, and extracts their hashtags. It then constructs a set of weighted and unweighted network structures based upon co-citation of hashtags within a tweet. The network diagrams are interpreted to have a set of clusters within them which represent 'conversations' that are happening in the pool of twitter messages. We track similarity between clusters from day to day to investigate how conversations develop.

References
These scripts depend upon a number of external utilities as listed below: import datetime print 'started at %s'%datetime.datetime.now() import json import gzip from collections import Counter from itertools import combinations import glob import dateutil.parser import pandas as pd import os import numpy as np import datetime import pickle import subprocess #load the locations of the various elements of the analysis with open('config.json','r') as jfile: config = json.load(jfile) print config

Data Files
We have twitter messages saved as compressed files, where each line in the file is the JSON object that the twitter sample stream returns to us. The files are created by splitting the streaming dataset according to a fixed number of lines -not necessarily by a fixed time or date range. A description of the collection process can be found in Appendix D.
All the files have the format posts_sample_YYYYMMDD_HHMMSS_aa.txt where the date listed is the date at which the stream was initialized. Multiple days worth of stream may be grouped under the same second, as long as the stream remains unbroken. If we have to restart the stream, then a new datetime will be added to the files.
# Collect a list of all the filenames that will be working with files = glob.glob(config['data_dir']+'posts_sample*.gz') print 'working with %i input files'%len(files) Its helpful to have a list of the dates in the range that we'll be looking at, because we can't always just add one to get to the next date. Here we create a list of strings with dates in the format 'YYYYMMDD'. The resulting list looks like: ['20141101', '20141102', '20141103', ... '20150629', '20150630']  The most data-intensive part of the analysis is this first piece, which parses all of the input files, and counts the various combinations of hashtags on each day.
In this demonstration we perform this counting in memory, which is sufficient for date ranges on the order of

Supporting Structures
Step 1: Count hashtag pairs weeks, but becomes unwieldy beyond this timescale. We'll save these in a very specific directory structure that will simplify keeping track of our data down the road, when we want to do more complex things with it. An example: We create a row for every combination that has a count of at least two.
In this code we'll use some of the iPython 'magic' functions for file manipulation, which let us execute shell Step 2: Create Weighted Edge Lists commands as if through a terminal. Lines prepended with the exclaimation point ! will get passed to the shell. We can include python variables in the command by prepending them with a dollar sign $ .
for key in tallydict. # we'll be reading a lot of files like this, # so it makes sense to create a function to help with it. def read_cluster_file(infile_name): """ take a file output from COS and return a dictionary with keys being the integer cluster name, and elements being a set of the keywords in that cluster""" clusters = dict ( We want to know how a cluster on one day is related to a cluster on the next day. For now, we'll use a bruteforce algorithm of counting the number of nodes in a cluster that are present in each of the subsequent day's cluster. From this we can get a likelihood of sorts for subsequent clusters. We'll define a function that, given the clusers on day 1 and day 2, creates a matrix from the two, with day1 clusters as row elements and day2 clusters as column elements. The entries to the matrix are the number of nodes shared by each cluster. #brute force, without the intra-day clustering def compute_transition_likelihood(current_clusters, next_clusters): transition_likelihood = np.empty([max(current_clusters.keys())+1, max(next_clusters.keys())+1]) for current_cluster, current_elements in current_clusters.iteritems(): for next_cluster, next_elements in next_clusters.iteritems(): #the size of the intersection of the sets transition_likelihood[current_cluster, next_cluster] = ( len(current_elements & next_elements) ) return transition_likelihood We want to compute transition matricies for all clusters with every k and every threshold. We'll save the matrix for transitioning from Day1 to Day2 in Day1's folder. In many cases, there won't be an appropriate date/threshold/k combination, so we'll just skip that case.