{'fraction_non_alphanumeric': 0.03355606138483002, 'fraction_numerical': 0.007526250091783537, 'mean_word_length': 4.689014202172097, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Stance Classification of Ideological Debates: Data, Models, Features, and Constraints

Determining the stance expressed in a post written for a two-sided debate in an online debate forum is a relatively new and challenging problem in opinion mining. We seek to gain a better understanding of how to improve machine learning approaches to stance classification of ideological debates, specifically by examining how the performance of a learning-based stance classification system varies with the amount and quality of the training data, the complexity of the underlying model, the richness of the feature set, as well as the application of extra-linguistic constraints.


Introduction
Determining the stance expressed in a post written for a two-sided debate in an online debate forum is a relatively new task in opinion mining. Given a post written for a two-sided topic discussed in an online debate forum (e.g., "Should abortion be banned?"), the goal of debate stance classification is to determine which of the two sides (i.e., for and against) its author is taking.
Previous approaches to debate stance classification have focused on three debate settings, namely congressional floor debates (e.g., Thomas et al. (2006), Bansal et al. (2008), Balahur et al. (2009), Yessenalina et al. (2010), Burfoot et al. (2011)), company-internal discussions (e.g., Agrawal et al. (2003), Murakami and Raymond (2010)), and online social, political, and ideological debates in public forums (e.g., Somasundaran and Wiebe (2010), Wang and Rosé (2010), Anand et al. (2011), Biran and Rambow (2011), Hasan and Ng (2012)). As Walker et al. (2012) point out, debates in public forums differ from congressional debates and company-internal discussions in terms of language use. Specifically, online debaters use colorful and emotional language to express their points, which may involve sarcasm, insults, and questioning another debater's assumptions and evidence. These properties could make stance classification of online debates more challenging than that of the other two types of debates.
Our goal in this paper is to gain a better understanding of how to improve machine learning approaches to stance classification of online debates by examining the following questions, which can be broadly categorized along four dimensions: Data. Can we improve the performance of a stance classification system simply by increasing the number of stance-annotated debate posts available for training? Note, however, that the number of stance-annotated posts that can be downloaded from debate forums for a given debate domain (e.g., Abortion) is fairly limited. A natural question is: given a debate domain, can we identify from different data sources a large number of documents where authors express viewpoints relevant to the domain (e.g., blog posts, news articles) and then stance-label them heuristically, with the goal of employing these noisily labeled documents as additional data for training a stance classifier? Features. The simplest kind of features one can think of is probably n-grams. Nevertheless, a stance classifier trained on unigrams is a relatively strong baseline (Somasundaran and Wiebe, 2010). Anand et al. (2011) augment an n-gram feature set with four types of features: document statistics, punctuations, syntactic dependencies, and, if applicable, the set of features computed for the immediately preceding post in the discussion thread (see Section 3 for details). How effective are Anand et al.'s features in improving an n-grambased stance classifier? Will adding semantic features improve performance further? Models. The simplest stance classification model is probably one that assigns a stance label to each debate post independently of the other posts. Can we get better performance by exploiting the linear structure inherent in a post sequence? Since a post may contain materials irrelevant to stance classification, can we train a better model by learning only from the stancerelated sentences without relying on sentences manually annotated with stance labels?
Constraints. Extra-linguistic inter-post constraints, such as author constraints (see Section 3), have been shown to be effective in improving stance classification performance by postprocessing the output of a stance classifier. Will the effectiveness of these constraints be dependent on the underlying debate domain? Will it be dependent on the accuracy of the stance classifier to which they are applied?
By examining these questions, we can potentially determine how the performance of a stance classification system varies with the amount and quality of the training data, the complexity of the underlying model, the richness of the feature set, as well as the application of extra-linguistic constraints. In our evaluation, we focus on stance classification of ideological debates.

Datasets
For our experiments, we collect debate posts from four popular domains, Abortion (ABO), Gay Rights (GAY), Obama (OBA), and Marijuana (MAR). Each post should receive one of two domain labels, for and against, depending on whether the author of the post supports or opposes abortion, gay rights, Obama, and the legalization of marijuana, respectively. To see how we obtain these domain labels, let us first describe the data collection process in more detail.
We collect our debate posts for the four domains from an online debate forum 1 . In each domain, there are several two-sided debates. Each debate has a subject (e.g., "Abortion should be banned") for which a number of posts were written by different authors. Each post is manually tagged with its author's stance (i.e., yes or no) on the debate subject. Since the label of each post represents the subject stance but not the domain stance, we need to automatically convert the former to the latter. For example, for the subject "Abortion should be banned", the subject stance yes implies that the author opposes to abortion, and hence the domain label for the corresponding label should be against. We construct one dataset for each domain. Statistics of these datasets are shown in Table 1.

Experimental Setup
In this section, we describe the experimental setup behind our investigation of the issues along the four dimensions of learning-based stance classification: models, features, data, and constraints.

Models
We seek to examine how model complexity impacts stance classification performance. We consider three types of models.
The first type of models is a binary classifier that assigns a stance label to each debate post independently of the other posts. We employ a generative model (Naive Bayes (NB) with add-one smoothing) and a discriminative model (Support Vector Machines (SVMs), as implemented in SVM light (Joachims, 1999)) in our investigation. This enables us to determine whether the relative performance of generative models and discriminative models changes with the amount of training data (Ng and Jordan, 2002), and whether generative models can handle complex, possibly overlapping features as well as discriminative models.
The second type of models, sequence models, is motivated by an observation: since a post in a post sequence is a reply to its parent post, its label should be determined in dependent relation to that of its parent. Consequently, these models assume as input a post sequence and output a sequence of stance labels, one for each post in the input sequence. As before, we employ two sequence labelers, one generative (first-order Hidden Markov Models (HMMs) with add-one smoothing) and one discriminative (linear-chain Conditional Random Fields (CRFs) (Lafferty et al., 2001), as implemented in Mallet (McCallum, 2002)).
The last type of models is fine-grained models. These models jointly determine the stance label of a debate post and the stance label of each of its sentences. We hypothesize that modeling sentence stances could improve document stance classification performance: for example, features computed from sentences with a neutral stance should not play any role in determining the document stance. To avoid the cost of hand-annotating sentences with stance labels for training a sentencestance classifier, we determine sentence stances in our model in an unsupervised manner. Moreover, while it is possible to implement fine-grained models based on NB, SVM, HMM, and CRF, we will focus exclusively on those based on NB and HMM. The reason is that they are easier to implement because we employ our own implementation of NB and HMM in these experiments.
The generative story. To generate a document d i , we first pick a document stance c with probability P(c). Given c, we generate each sentence in d i independently of each other. To generate a sentence e m , we first pick a sentence stance s with probability P(s|c), and then generate f n , the value of the nth feature representing e m , with probability P(f n |s, c).
A few points deserve mention. First, finegrained NB and fine-grained HMM both employ this document generative story, differing only in terms of whether the document stance is generated independently (NB) or in dependent relation to that of the preceding post (HMM). Second, while a document stance can have one of two possible values (for and against), a sentence stance can have one of three possible values (for, against, and neutral). Note that the stance of a sentence can be expressed by someone other than the author of the post; for example, a sentence may restate an opposing opinion by a different author.
Training the fine-grained models. As noted above, we need to estimate P(c), P(s|c), and P(f n |s, c). P(c) can be estimated from the stancelabeled training documents. 2 However, since sentence stances are hidden, we estimate P(s|c) and P(f n |s, c) using the Expectation-Maximization (EM) algorithm (Dempster et al., 1977).
To employ EM, we begin by heuristically labeling each sentence with a stance as follows. First, for each document stance c, we identify the list of informative unigrams, which consists of the openclass words that appear at least 10 times in the training data and is associated with c at least 70% of the times. Then, given a sentence e m , its stance label is determined by taking a simple majority vote using the stance labels associated with the informative unigrams appearing in e m . In case of a tie, e m is labeled as neutral. 3 After heuristic labeling, we begin with the Mstep, where we estimate model parameters P(s|c) and P(f n |c, s) from the training data. Since the data is now stance-labeled at both the document and sentence level, we can estimate these parameters using maximum likelihood estimation.
Next, we proceed to the E-step, where the goal is to estimate P(s|e m , d i , c), the probability that a sentence expresses a sentence stance given the document stance. From the above generative story, where F is the set of features in sentence e m . We run EM until convergence.
Applying the fine-grained models. After training, we can apply the fine-grained models to classify each test post d i . For fine-grained NB, we employ the following equation: where S(d i ) is the set of sentences in test post d i , and s max is the sentence stance with the maximum conditional probability (obtained using Equation 1) for sentence e m in d i . For fine-grained HMM, we employ Viterbi to decode a post sequence, using P(d i |c) as the "output probability" of a test post given stance c.

Features
We seek to examine how the features used to train the stance classification system impact its performance. We consider three feature sets.
N-gram features. The first feature set consists of unigrams and bigrams collected from the training posts. We encode them as binary features that indicate their presence or absence in a given post.  Their ngram features include both the unigrams and bigrams in a post, as well as its first unigram, first bigram, and first trigram. The features based on document statistics include the post length, the number of words per sentence, the percentage of words with more than six letters, and the percentage of words as pronouns and sentiment words. The punctuation features are composed of the repeated punctuation symbols in a post. The dependencybased features have three variants. In the first variant, the pair of arguments involved in each dependency relation extracted by a dependency parser is used as a feature. The second variant is the same as the first except that the head (i.e., the first argument in a relation) is replaced by its part-ofspeech (POS) tag. The features in the third variant, the topic-opinion features, are created by replacing each feature from the first two types that contains a sentiment word with the corresponding polarity label (i.e., + or −).
Adding frame-semantic features. To provide semantic generalizations, we create features computed using FrameNet semantic frames (Baker et al., 1998). More specifically, we first apply SE-MAFOR (Das et al., 2010) to create a framesemantic parse for each sentence in a given debate post. Then, for each frame that a sentence contains, we create three types of frame-semantic features, as described below.
A frame-word interaction feature is a binary feature composed of (1) the name of the frame f from which it is created, and (2) an unordered word pair in which the words are taken from two frame elements of f . Specifically, for each pair of frame elements f e 1 and f e 2 of a frame f , we cre-ate one frame-word interaction feature from each unordered word pair composed of one word from f e 1 and one word from f e 2 . Consider the framesemantic parse of the sentence Every woman has the right to choose abortion shown in Table 2. Given the frame Possession and its frame elements Every woman and the right to choose abortion, we can generate frame-word interaction features such as Possession-right-woman, Possession-choosewoman, Possession-abortion-woman.
A frame-pair feature is a binary feature composed of a word pair corresponding to the names of two frames, in which the target of the first is present in a frame element of the second. Specifically, for each frame element f e of a frame f , if a substring of f e is the target of a frame f 2 , we create a frame-pair feature composed of the ordered pair (f 2 , f ). Consider the example in Table 2 again. Given the frame Possession and its frame elements Every woman and the right to choose abortion, we can create three frame-pair features, People:Possession, Choosing:Possession, and Correctness:Possession, since woman, choose, and right are the targets of frames People, Choosing, and Correctness, respectively.
A frame n-gram feature is the frame-based version of a word n-gram feature. Given a word unigram or bigram in which each word is an openclass word, we create all possible frame n-gram features from it by replacing one or more of its words with its frame name (if the word is a frame target) or its frame semantic role (if the word is present in a frame element). For instance, in the word bigram woman+has from the sentence in Table 2, both woman and has are open-class words and are targets of People and Possession, respectively. Hence, we create for woman+has three frame n-gram features: woman+Possession, People+has, and People+Possession. In addition, since woman plays the role of Owner in Possession, we create two more frame n-gram features, Owner+Possession and Owner+has. Using the frame-semantic features. One way to use the frame-semantic features is to incorporate them into Anand et al.'s (2011) feature set and train a stance classifier on the augmented feature set. 4 We employ a different way of using the frame-semantic features, however. We train two  stance classifiers, C A and C F S . C A is trained using Anand et al.'s (2011) features, whereas C F S is trained using only the frame-semantic features. After training, we use the classifiers to predict the stance for a post x in the test set as follows. We first apply them independently to classify x, and then predict the stance for x by linearly interpolating the resulting classification values. The value of the interpolation constant is tuned to maximize performance on development data. 5

Data
We seek to examine how the amount and quality of the training data impact stance classification performance.
To determine how classification performance varies with the amount of training data, we will plot learning curves in our evaluation.
As far as training data quality is concerned, our goal is to collect documents discussing viewpoints relevant to the debate domain of interest from different sources (e.g., blogs, news websites), stancelabel them heuristically, and determine how these noisily labeled documents can be used in combination with the stance-annotated debate posts to train a stance classification system. Below we describe how we collect and utilize these documents.
Collecting noisily labeled documents. To collect noisily labeled documents, we employ a twostep procedure. We (1) create using commonsense knowledge a list of phrases that are reliable indicators of both stances for each domain; and then (2) use each phrase as an exact search query to retrieve noisily labeled documents from the Web.
Sample phrases that we create for each stance of each domain are shown in Table 3. 6 For instance, for the Abortion domain, the phrase I support abortion indicates the author's support for abortion. In contrast, I think abortion should be banned is indicative of the author's stance against abortion. Since we use each phrase as a search query in the second step, we manually paraphrase each of them in hope to increase the number of retrieved documents. For instance, we create for abortion should be banned paraphrases such as abortion should be prohibited, abortion should be illegal, and abortion should not be allowed. Some paraphrases are created simply by employing different forms of a proper noun (e.g., Obama, Barack Obama, and President Obama). Table 4 shows the statistics of the noisily labeled documents. It took us less than two person-days to create the phrases and their paraphrases for each domain. Roughly the same number of phrases were created for the two stances in a domain.
As noted above, we use each phrase created in the first step as an exact search query to retrieve documents from the Web using Bing's Search API. 7 A closer inspection of the retrieved documents reveals that many of them contain materials irrelevant to the search query. One of them, for instance, is a blog article discussing different facets of women rights, followed by comments from several readers. The search query that retrieved the document appeared in one of the readers' comments. In this case, it makes sense to delete everything but this reader's comment from the document before using it as noisily labeled data. For this reason, we heuristically extract the portion of each retrieved document that is relevant to the search query. More specifically, we define the relevant portion of a document as the smallest string that contains the search query string and is delimited by HTML tags. Note that we discard documents that contain less than 10 words (in order to avoid documents with no useful content) or are retrieved from www.createdebate.com (in or-  der to avoid overlaps with our evaluation datasets).

Training with noisily labeled documents.
Given these noisily labeled documents, how can they be used in combination with the (cleanly labeled) debate posts in the training set for training stance classifiers? Motivated by Nguyen and Moschitti (2011), we train two stance classifiers, C c and C c+n . C c is trained on only the debate posts in the training set. C c+n , on the other hand, is trained on both the debate posts and the noisily labeled documents. 8 Both of them use the same set of features. After training, we use these classifiers to predict the stance for a post x in the test set as follows. We first apply them independently to classify x, and then predict the stance for x by linearly interpolating the resulting classification values. The value of the interpolation constant is tuned to maximize performance on development data. 9

Constraints
Previous work on stance classification of congressional debates has found that enforcing author constraints (ACs) can improve classification performance (e.g., Thomas et al. (2006)). ACs are a type of inter-post constraints that specify that two posts written by the same author for the same debate domain should have the same stance, and are typically used to postprocess the output of a stance classifier. We seek to determine how ACs impact the performance of a system for stance-classifying ideological debate posts, and whether their effectiveness depends on the debate domain.
In our experiments, we enforce ACs as follows. We first use a stance classifier to classify the test posts. Note that the classification value of a post can be thought of as a probabilistic vote that a post can cast on the stance labels. Then, given a set of test posts written by the same author for the same debate domain, we sum up the probabilistic votes cast by these posts, and assign to each of them the stance that receives the larger number of votes.

Evaluation
In the previous section, we described the experimental setup for investigating the issues pertaining to the four dimensions of learning-based stance classification. In this section, we begin by describing the general experimental setup and then report on and discuss the evaluation results.

General Experimental Setup
Results are expressed in terms of accuracy obtained via 5-fold cross validation, where accuracy is the percentage of test instances correctly classified. Since all experiments require the use of development data for parameter tuning, we use three folds for model training, one fold for development, and one fold for testing in each fold experiment. All SVM and CRF learning parameters are set to their default values in SVM light and Mallet, respectively.
Learning curves are generated for all the experiments. Each point on a learning curve is computed by averaging the results of five independent runs corresponding to five different randomly selected training sets of the required size. To ensure a fair comparison of different learning models, the same five randomly selected training sets of the required size are used to train the models. Since the models based on HMMs and CRFs need to be trained on post sequences, we assemble a training set of a given size as follows: whenever a post is sampled for inclusion into the training set, we incorporate all the posts in the same post sequence into the training set.

Results
Results for the four domains are shown as four sub-tables in Table 5. Owing to space limitations, we do not show the learning curves. Rather, we show results for three selected points on each learning curve, which correspond to the three major columns in each sub-table. For instance, for Abortion, the three selected points correspond to training set sizes of 300, 600, and 1000. Within each major column there are six columns corresponding to the six learning models, among which the two fine-grained models are marked with the  To determine statistical significance, we conduct paired t-tests (p < 0.05). These significance tests can be divided into three groups. The first group aims to determine whether the performance difference between the two systems shown in consecutive rows in a given column is statistically significant. If a number is marked with a tagger ( †), it means that the performance difference between the corresponding system and the one in the previous row is statistically significant. The second group aims to determine whether the performance difference between two learning models are significant. We tested significance for three pairs of learning models: (1) SVM and NB; (2) NB and HMM; and (3) HMM F and CRF. If a number is marked with an asterisk ( * ), it means that the performance difference between the corresponding learning model and the one in the same pair is statistically significant. 10 The third group aims to determine whether the performance difference between NB/HMM and the corresponding fine-grained version of the model is statistically significant. If a number for a fine-grained model (NB F , HMM F ) is marked with a double dagger ( ‡), it means that the performance difference between the model and its corresponding coarse-grained version (NB, HMM) is significant.

Discussion
Q: Can we improve performance by increasing the number of stance-labeled posts in the training set? A: Yes. Keeping other factors constant, as we increase the number of (cleanly labeled) training posts from 100 to 500, we see significant improvements on all four domains: accuracies increase by 1.5, 2.4, 2.0, and 3.1 points for ABO, GAY, OBA, and MAR, respectively. As we further increase the number of training posts from 500 to 1000, we see another significant rise in performance: accuracies improve by 2.7 and 1.3 points for ABO and GAY, respectively. For ABO, GAY, and OBA, increasing the training set size seems to have a more positive impact on systems employing a simple feature set (W) than on those employing richer feature sets. Other than that, the degree of improvement does not seem to be dependent on the complexity of the model and the richness of the feature set. Q: Which model is better, NB or SVM? A: There is no clear winner. Other factors being equal, SVM beats NB significantly in 17% of the cases, NB beats SVM significantly in 27% of the cases, and the two are statistically indistinguishable in the remaining cases. Neither generative models nor discriminative models seem to have an advantage over the other for this task.
Q: Are the sequence models better than their nonsequence counterparts? A: Yes. Comparing NB and HMM, we see that HMM consistently outperforms NB significantly, with improvements ranging from 1.6 to 2.2 points for the four domains. Now, comparing HMM and CRF, we see that while CRF does not always perform significantly better than HMM, in no case does it perform significantly worse. 11 Taken together, both sequence learners perform significantly better than NB. Since NB and SVM perform at the same level, we can conclude that sequence models indeed offer better performance.

Q:
Are the fine-grained models better than their coarse-grained counterparts? A: Considering HMM and HMM F , the answer is yes: HMM F beats HMM significantly by 1.1 to 2.1 points for the four domains. Considering NB and NB F , the answer is mostly yes: NB F beats NB significantly by 1.2 to 2.3 points for GAY and OBA respectively. For the remaining domains, NB F performs significantly better than NB in most cases, especially when the n-gram feature set and the Anand et al.'s feature set are used.

Q:
Which is the best model? A: HMM F and CRF achieve the best results, but there is no clear winner between them. Other factors being equal, CRF beats HMM F significantly in 26% of the cases, HMM F beats CRF significantly in 21% of the cases, and the two are statistically indistinguishable in the remaining cases.