{'fraction_non_alphanumeric': 0.030392204158425393, 'fraction_numerical': 0.012301606445076945, 'mean_word_length': 4.665482372232851, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Bootstrapped Model to Detect Abuse and Intent in White Supremacist Corpora

Intelligence analysts face a difficult problem: distinguishing extremist rhetoric from potential extremist violence. Many are content to express abuse against some target group, but only a few indicate a willingness to engage in violence. We address this problem by building a predictive model for intent, bootstrapping from a seed set of intent words, and language templates expressing intent. We design both an n-gram and attention-based deep learner for intent and use them as colearners to improve both the basis for prediction and the predictions themselves. They converge to stable predictions in a few rounds. We merge predictions of intent with predictions of abusive language to detect posts that indicate a desire for violent action. We validate the predictions by comparing them to crowd-sourced labelling. The methodology can be applied to other linguistic properties for which a plausible starting point can be defined.


I. INTRODUCTION
Intelligence analysts scan online data looking for the signals of those who plan to carry out violent attacks. Several models have been developed for what might be called "violent thought" in islamist and white supremacist forums. Detecting when violent thought transforms into violent action is more problematic. There are many examples of so-called "armchair jihadists" who post extensively but never do anything; and conversely those who move very rapidly to violent action without extensive discussion (for example, Farhad Khalil Mohammad Jabar, who killed an Australian civilian police employee apparently within hours of hearing a radical islamist talk).
Models that detect abusive language and hate speech are moderately well developed. We expand their power by adding the capability to detect intent. Intent can be defined as "the state of mind of one who aims to bring about a particular consequence" [8]; when tied to abusive language, this acts as a signature for those who are most likely to carry out violent actions. Intelligence analysts can use this abusive intent model to focus attention on those whose posts make them of greatest concern.
Datasets labelled by intent do not exist (except perhaps in classified environments). We design a bootstrapped approach that starts from small, widely-agreed signals of intent from the literature; and then bootstraps these into a pair of models, one using n-grams and one using biLSTMs, to predict intent.
These are coupled with a deep-learning model of abusive language to label posts in White Supremacist settings by their abusive intent. The intent model's predictions were compared to labels generated by human volunteers, with more than 80% agreement.

II. RELATED WORK
Linguistic understanding of intent begins with the work of Leech [17] who observed that will and going to are the strongest signals of the "future as outcome of present" [16]. However, a distinction must be drawn between the future as an outcome of present circumstances, or present intentions, and so the author's stance with respect to these verbs is critical. In particular, the presence of first-person pronouns distinguishes intent from observation. Frame analysis examines the effect of communication on audiences, and has been used by Sanfilippo [21] to assess the likelihood of violent action from online posts. This approach has been partially automated [22].
Intent has also been considered as a social process. The relationship between an in-group and an out-group has been leveraged to detect abusive language, and also to understand and characterise threats [26]. The cooccurrence of abusive language and intent has been observed to be associated with violent actions [1,18,24,25].
Detection of properties of interest in natural language either uses bag-of-words approaches, or deep learning, mostly using biLSTMs. Intent detection has previously been studied in the context of business interactions: what is this (potential) customer trying/planning to do [3,4,9,10,14,28]. Others have built models to detect intent in wider contexts such as forums using patterns such as actionable verb object [27]; or regular expressions I . . . want . . . to . . . [20]. The definition of intent used by these approaches is quite broad.
The detection of abusive language has been well-studied [5,6,15,29] using both bag-of-words and deep-learning approaches, with typical accuracies in the 90% range.
A technique on which we build, called double bootstrapping, was developed by Gao et al. [7]. For unlabelled datasets, it begins from a small dictionary of hate-speech terms, generates initial labels for the data, feeds these into two models, which then co-train, each learning from the labels, making new predictions, and then passing the predictions to the other model.

III. METHODOLOGY
Three White Supremacists forums were used as datasets: Stormfront, Iron March, and the manifesto of the New Zealand attacker in 2019. As well, a Wikipedia dataset was used as a contrast set, and an abusive language dataset used to train a predictor for abuse.
Forum posts are noisy natural language, with misspellings and typos, in-group language, and non-textual elements such as emojis. Each dataset was processed to remove quotations (detected from html tags), user handles, html tags, emojis, and Unicode characters. Hashtags were replaced by their content, divided into words if camel case was used; characters repeated more than twice were replaced by two occurrences. Table I shows the effect of preprocessing. The reduction in size for Stormfront reflects the large number of quotations typically used, both from the other posts and from media web sites. Documents were split into segments, broken at sentence boundaries or semicolons. Segments are the primary units for which predictions of intent and abuse will be made.
A segment by word n-gram frequency matrix was created, using n = 3 to 6. Examination of the cumulative frequency indicated that almost nothing was lost by taking only the 500,000 most common n-grams.
Embeddings for the data from Stormfront were created using FastText 0.9.2 [2] to produce 200-dimensional vectors, using skipgram with default training parameters. The training process took roughly 5 days.
Deriving an embedding from the Stormfront dataset, rather than using the generic embedding, shows the importance of in-group language patterns. Table II shows the neighbours of "liberal" in the generic FastText embedding, while Table III shows the neighbours in the embedding derived from Stormfront. The default embedding shows a conventional political view of "liberal" while the customised embedding shows a much more doctrinaire view.

A. Intent label inference
Datasets labelled with intent are not publicly available, so we develop a way of inferring intent labels by bootstrapping. The first step is to develop a template for expressing intent. The basic structure of the template is: first-person pronoun, desire verb ("going to", "will"), action verb ("fight") and optionally a target and timing. The template elements must  Initial labels for each segment are generated as follows: if it contains a match for the template, it is labelled +1; if it contains a negation question, or a second-or third-person  pronoun, it is labelled 0; otherwise it is labelled 0.5. A set of segments from Wikipedia was included in the set as well, all labelled with 0. Wikipedia's style guidelines disallow instances of first-person intent [30] so this enhances the dataset with examples of (highly likely to be) non-intent segments. The desire verbs in the template can be considered as varying from weak ones (dreaming of -"'I am dreaming of attacking them") to strong (have to -"I have to attack them"). In the initial round, any verb that is used in the desire verb slot is counted as a desire verb, although the obvious choices -"going to", "will" -were by far the most common.
The initial label generation phase, applied to the segments from Stormfront and Wikipedia labelled 1.4% as intentful, 29.1% as non-intentful, and 69.5% as undetermined. As expected, all of the Wikipedia segments were labelled as nonintentful.
The set of desire verbs is now refined. This requires both expansion so that misspellings and other variants are captured; and restriction so that only strong desire verbs remain. Fortunately, using the geometry of the embedding space solves both problems.
The process is initialised from a seed set of "obvious" desire verbs: "want", "need", "going", "have", "about", "planning", and "will". The embedding is treated as a vector space and verbs derived hypercone prism based on the seven initial vectors are treated as a strong desire verbs. To be included, the unnormalized cosine similarity of a selected desire verb must be less than twice the distance to the mean of the seed set than at least one member of the set. This produces 596 desire verbs, including verbs such as "seek" and "aiming". 95.3% of the segments initially labelled as intentful still qualify.  The model is based on two learners: an n-gram based learner; and a deep learner that uses the FastText embeddings and a BiLSTM-attention architecture [23,31]. These are used in parallel, and the results of each better inform the signals of intent for the next round.
1) n-gram learner: Assume that we have a set of label values between 0 and 1 for the segments; (initially this is obtained from the template matching). The n-gram predictor first assigns a score to each n-gram based on the ratio of how often it occurs in segments whose label is greater than 0.5 versus how often it occurs in segments whose label is less than 0.5, both frequencies normalized by the number of such segments in which it is present. n-grams in the 99.9th percentile of ratio values in either direction are considered predictive of intent or non-intent.
Segments are labelled as intent if they contain only intentful n-grams; as non-intent if they contain only non-intentful ngrams, but unlabelled if both or neither happen to be present.
2) Deep learner: In parallel, a deep learner using a biL-STM is applied to the segments, knowing the FastText embeddings of each word. The network architecture is shown in Figure VI. The biLSTM begins with a conventional learning phase in which it trains based on a randomly selected set of segments with the labels for each segment rounded to 0 or 1). As it trains the loss contribution from each segment is downweighted by how far its current label is from either 1 or 0.
After this phase, the deep learner then predicts labels for all of the segments, including those without labels. Segments close to the extremes have their scores altered by 10% to make them even stronger.  3) Merging predictions for the next round: Both learners are trying to increase the number of strong predictions, that is predictions close to 1 or close to 0. However, they are simultaneously learning the natural language signals of intent. To prevent too rapid convergence of labels, each model is limited in how many labels it can present to the merge mechanism.
If the previous round of each model had predicted n p labels with value 1 and n n labels with value 0, then at most these numbers of further records can be suggested to the merge mechanism. In other words, if 100 labels were 1 in the previous round, then at most 200 segments can be presented as 1 to the merge mechanism in the current round.
This limitation on convergence could potentially slow down the overall process when a model quickly discovers the required signals, However, this does not seem to be the case for complex properties such as intent.
The merge mechanism relabels each segment based on the label from either of the models, except if they have contradictory senses (that is, one predicts intent while the other predicts non-intent) in which case the label remains unchanged. If a segment is labelled either 0 or 1, that label is locked and will not change in subsequent rounds.
The number of rounds can be altered for each particular dataset, but 5 or 6 seems to be adequate.

IV. MODELLING ABUSIVE LANGUAGE
There is intrinsic interest in modelling intent, but the applications we envisage are those where intent per se is not interesting ("I'm going to buy her a birthday present") but where intent associated with abusive language signals the potential for violent action. We must therefore also identify segments that express abuse.
Abusive language detection is a well-studied problem and labelled data is available [15,31]. We use three relevant datasets: a small set of documents from Stormfront; a corpus from a competition run by Impermium for detecting insults [12]; and a dataset from a competition by Conversation AI to identify multiple types of abusive language in Wikipedia comments [13]. The documents from these datasets were randomly mixed to ensure no sequential structure, creating a dataset with 240, 846 samples (Table VII). A biLSTM predictive model, using the word embeddings derived from Stormfront and the same network as for intent (Table VI).
An 85-15 train-test split (204,719 training segments, 36,127 test segments) was used, with a maximum of 50 epochs, and early stopping with patience 3. The learning rate was 0.001, beta one and two were 0.9 and 0.999, and epsilon was 1 × 10 −7 . This model's accuracy was 86.7%.

A. Combining intent and abuse scores
Both models produce predictions in [0, 1] and there are multiple ways in which these could be combined. For our application domain, where the primary goal is to identify segments that are high in both intent and abuse, we multiply the two predicted scores. Even moderate scores in either dimension reduce the overall score, focusing attention on segments of greatest practical interest.

B. Scores for documents from scores for segments
Combining scores for segments to produce a single score for each document raises subtle problems. An author may be abusive in one segment and segue into intent in the adjacent one, or vice versa. A document may begin with a discussion of a perceived problem (neither abusive or intentful) and only then begin to express abuse or intent or both.
The approach we chose was to consider the abuse and intent scores of each segment separately, take each set of three adjacent (overlapping) segments in a document, take the maximum abuse score and the maximum intent score in the set of three, and then form the product of these maximums. The overall score for the document is the maximum of these products over all of the three-segment windows.

V. PERFORMANCE AND VALIDATION
In each epoch of intent training, the sequence learner and the deep learner are run in parallel, and then a consensus is derived. Table VIII shows some of the highest scoring intentful sequences after 6 epochs of training. In general, both the set of strongest intentful and non-intentful sequences converge quickly.  Figure 3 shows the convergence of the deep learner for intent. This figure should be interpreted with care because the labels are changing from one epoch to the next, but the convergence is clear.
The effect of each epoch on the label consensus is shown in Figure 4. The set of high-intent segments do change, but not enough to be visible at the top of the figure; the non-intent segments rapidly increase. Figure 5 shows the convergence of the deep learner for abusive language.
Tables IX, X and XI show the highest ranking segments sorted by predicted intent score, with the corresponding abuse and abusive-intent scores shown as well. Unsurprisingly in these contexts there is considerable correlation between intent and abuse. Figure 6 shows Shapley values for the individual words in an example, illustrating which words have an impact on the predictions. The abusive language model identifies "kill you"

A. Comparison with human judgements
Since the labelling process is inductive, we use agreement with human assessments as a validation technique. A web site was created in which volunteers could label segments as intentful or not, and abusive or not. Each volunteer received tranches of 5 segments, plus a qualifying example (not from the dataset and constructed with a known label). Tranches in which a user answered incorrectly to the qualifying question were discarded. No volunteer received more than 30 examples (6 tranches) to label.
The samples shown to volunteers were sampled from the extremes of the labelling, that is segments whose labels were either in [0, 0.4] or [0.6, 1]. A total of 5000 segments were randomly selected and randomly presented to participants. Each segment was scored on a first to 3 basis (that is, 3 consistent votes up to a maximum of 5 total votes) and removed from the candidate set when it had received 3 consistent votes. User predictions were scored as binary (majority intentful or nonintentful) and weighted (the ratio of votes for intentful to total votes).  The agreement with the computational predictive model was 80% on a binary basis, and 81% on a weighted basis (that is, comparing weighted human labelling to real-valued intent labels). This suggests that some examples that the intent predictor found difficult were also ones on which human raters disagreed.
Inter-rater agreement above 70% are normally considered adequate, so these results support that the intent predictor is performing well, especially as intent is a subjective category  The confusion matrix between model and human predictions is shown in Figure 7. Raters found considerably more intent than the predictive model did. Tables XIII and XIV show examples where the model and human raters disagreed. In some cases the human raters considered segments with past tenses as conveying intent ("I decided to join", "I wanted to feel"). In other cases, they saw what is arguably a weak form of intent ("I may well respond") which the model does not see. Phrases such as "it going to be splat time" do not contain any linguistic signals of intent but were plausibly considered to be (at least potential) intent.
For the examples where the model detected intent but the human scorers didn't, it is clear that humans do not consider "we need" or "we want" to be expressing intent. However, "we continue to kick" and "we are going to be labelled" are clearly mistaken intent prediction by the model. These divergences suggest that the model could be improved by paying more attention to first-person plural pronouns, since the human scorers clearly believe that these are weaker signals of intent: "I need to" is regarded as much stronger than "we need to".

VI. CONCLUSIONS
We address the problem of detecting posts that convey both abusive content and the intent to act, and so are targets for intelligence analysts. We build a predictive system that infers signals of intent from unlabelled data and a small seed set, using both an n-gram and a deep-learning approach, acting as colearners. The merged labels become stable within only a few rounds. We combine this with a deep-learning biLSTM for abusive language, and design ways to score individual segments on intent, abuse, and abusive intent; and score entire documents based on the scores of their segments. We appeal to face validation by showing the highest-scoring segments, and compare the predictions of the model with human scorers, achieving an agreement of 80% for class labels and 81% for regression.
The methodology developed here can be applied to any other linguistic property for which an appropriate seed set and template can be defined.     or what the f is being retarded and bothering me 0.912 0.00 jew watch jewish atrocities slave trade jewish slave ship owners quote for decades the white people of america have been subjected to a continual barrage from blacks and others that you and i are somehow responsible for the african slave trade and that we need to atone for our guilt 0.911 0.00 we need to be in the middle east like we need a second asshole 0.605 0.00 we can speculate all we want except for the cases of obese self esteem issues women but i m convinced there s something important here we re missing along the lines of why well to do pretty white girls lived abusive lives with charles manson and why hetero women with sexual trauma abuse in their pasts become strippers and porn starts and turn bisexual or lesbian 0.892 0.00 we need that here but the problem is getting people off their asses 0.891 0.00 we need a virus that attacks jews and non whites 0.931 0.40 com but i want to make sure i dont sound like a dumbass 0.933 0.25 more like future rapist car thief murderer rapist murderer either way prison or welfare we will support them