{'fraction_non_alphanumeric': 0.028799433664768156, 'fraction_numerical': 0.010876210702448756, 'mean_word_length': 4.978838014621008, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
"Read My Lips": Using Automatic Text Analysis to Classify Politicians by Party and Ideology

The increasing digitization of political speech has opened the door to studying a new dimension of political behavior using text analysis. This work investigates the value of word-level statistical data from the US Congressional Record--which contains the full text of all speeches made in the US Congress--for studying the ideological positions and behavior of senators. Applying machine learning techniques, we use this data to automatically classify senators according to party, obtaining accuracy in the 70-95% range depending on the specific method used. We also show that using text to predict DW-NOMINATE scores, a common proxy for ideology, does not improve upon these already-successful results. This classification deteriorates when applied to text from sessions of Congress that are four or more years removed from the training set, pointing to a need on the part of voters to dynamically update the heuristics they use to evaluate party based on political speech. Text-based predictions are less accurate than those based on voting behavior, supporting the theory that roll-call votes represent greater commitment on the part of politicians and are thus a more accurate reflection of their ideological preferences. However, the overall success of the machine learning approaches studied here demonstrates that political speeches are highly predictive of partisan affiliation. In addition to these findings, this work also introduces the computational tools and methods relevant to the use of political speech data.


Introduction
It is commonly believed that politicians lie, and do it often. However, existing political science literature on promise-keeping is more mixed than that adage would suggest. Both on specific issues and generally, political parties are surprisingly trustworthy, e.g., Pétry and Collette (2009). Early work in this area manually examined party platforms and similar documents to derive a list of promises and cross-referenced them with policy implementations, as in Bradley (1969) or Budge and Hofferbert (1990). Such analysis even entered the journalistic mainstream-PolitiFact kept an "Obameter" hand-tracking President Obama's promise-keeping (http://www.politifact.com/truth-o-meter/promises/obameter/). Automated text analysis techniques provide a new set of tools with which to address these questions. Recently,  and Gentzkow et al. (2016) have used these modern methods to quantify political polarization by extracting features from speeches 3 given in the US Congress. The work presented here builds on these early contributions by examining the value of political speech for predicting partisan affiliation. This approach allows us to directly test whether politicians' speech, and not just their votes, is an accurate indication of their policy positions.
A key theoretical difference between speeches and votes is that they represent two different levels of commitment. While speeches are public, they are nonbinding and are often treated as poorly predictive. On the other hand, votes, more specifically US Congressional roll-call votes, represent recorded positions taken on legislation that have an influence on policy. These votes can generally be treated as representing politicians' revealed preferences because of their higher commitment cost. However, given that many votes pass by large margins, the possibility of performative votes-those taken to showcase a position, not influence the outcome-mean that roll-call votes are not necessarily a perfect measure of true preferences. Moreover, there may be selection bias in terms of which topics are brought to a roll-call vote, as discussed by Carrubba et al. (2006). Automated text analysis, as discussed in this work, provides a new paradigm to address these challenges. By using methods from machine learning to predict political party from political texts, this work achieves two ends. First, since party membership is recorded and objective, it validates the accuracy of various classification methods. Second, classification by party replicates, in a simplified sense, voters' experience, since voters attempt to determine a candidate's ideology from available information. While party-blind elections are uncommon, they do exist, especially for judicial nominees, e.g., Bonneau and Cann (2013) and Burnett and Tiede (2014). Furthermore, primaries can be seen as "party-blind" elections in which voters face the task of choosing the more liberal or conservative candidate without party labels. Fitting the same classification models to ideology rather than party provides a more accurate simulation, though without the direct verifiability of partisan classification.
This work shows that the text of political speeches is indeed highly predictive of party affiliation. Using four types of machine learning models, we achieve classification accuracy in the 70%-95% range depending on the model and the data used to train it. These successful results lead to three main conclusions. First, they show that political speeches are valuable 4 for determining politicians' broad ideological positions. Second, they provide large-scale quantitative support for earlier works which manually analyzed the concordance of political speech and political action. Finally, the differing accuracies of the models used here clarifies which directions of refinement are likely to be most successful in providing even stronger results regarding classification or ideology identification from political speech.
Section 2 summarizes closely related literature. Section 3 describes the data used for this work, and Section 4 presents the text analysis approaches. Section 5 analyzes numerical results and Section 6 provides conclusions and directions for future research. A data excerpt and description of data preparation are included in Appendix A; details of the machine learning methods used are in Appendix B; and tables and figures appear in Appendix C.

Related Literature
There is ample literature on the trustworthiness of political parties. On the theoretical side, Aragonès et al. (2007) develop a model in which purely ideological candidates adhere to campaign promises under threat of punishment by voters. This argument holds up empirically; Pétry and Collette (2009) find in their meta-analysis of studies on political trustworthiness that parties keep 67% of their promises, albeit with wide variation. More germane to this paper are specific analyses using party-platform data to analyze trustworthiness in the US. Bradley (1969), Royed and Borelli (1997), and Budge and Hofferbert (1990) all manually check promise-keeping in political documents and argue that most salient promises are kept. Elling (1979) argues that differences in the language party platforms use are also predictive of trustworthiness. Despite some ambivalence as to the degree of the effect, there is broad agreement that party platforms predict political action.
Because they are recorded and publicly available, roll-call votes have become the primary basis for voting-based measures of political ideology. However, there is some literature on the inaccuracy of these votes as predictive measures. Hug (2010) argues that there is selection bias in which votes are recorded as roll-call votes in the Swiss Parliament and Carrubba et al. (2006) made a similar argument using European Parliament data, though they find a weaker selection bias. These works note, however, that in US Congress every vote is a 5 recorded roll-call vote, so selection bias is vastly less prominent if it is present at all.
In its raw form, roll-call voting behavior does not provide a clear picture of individual politicians' revealed preferences. Poole and Rosenthal (1985) provide the seminal work in the field of aggregating these votes to generate numerical estimates of political ideology. Their work generates what are known as Poole-Rosenthal DW-NOMINATE scores, henceforth DWN scores, the standard in the economics and political science literature. DWN scores are based on the idea that politicians' utility functions provide them with an "ideal point" in Euclidean space that describes their preferences and that politicians with more similar voting records have closer ideal points.
The treatment of political speeches as data composed of words rather than units of discourse is a relatively new approach. Laver et al. (2003) developed the pioneering Wordscores model to compare texts with unknown positions to those with a priori known positions. Slapin and Proksch (2008) develop Wordfish, a time-series version of the Wordscores model. These works use a variety of text sources; however, due to its size and richness, the US Congressional Record has become common both as a testing ground for new models and as a topic of research in and of itself. For instance, Quinn et al. (2010) determine clusters of speeches that are about related issues, find words that characterize each cluster, and predict speeches' location in these clusters. Jensen et al. (2012) follow a similar approach, extracting particularly partisan "trigrams," or sets of three words, then using the frequency of these phrases to quantify partisanship. Lauderdale and Herzog (2016)'s analysis is the closest to the goals of this paper. Using data from the Irish Daíl and US Congressional Record, they first divide the text into "debates," or sets of speeches on particular bills. They then use the Wordfish algorithm to determine the ideological position of speakers in each individual debate and determine ideological positions based on a common scaling of the Wordfish scores from each debate. Gentzkow et al. (2016), like Jensen et al., look at trends in political polarization, but, to reduce finite-sample bias, use a Bayes' Rule-based model for two-word "bigrams" to assign the probability that speakers belong to a given party.
Most recent work in text-as-data analysis of political speeches has a two-pronged goal.
First, progress in mathematics and computer science is used to develop models that more accurately classify or describe text in ways that align with observed phenomena, such as 6 political polarization. Second, these new models are used to draw conclusions about these real-world phenomena by performing analyses that exploit the fine-grained, quantitative measures of polarization or ideology recovered from the model. This w ork focuses on that second goal by using machine learning models for inspiration and examining the predictive value of political speech directly rather than as a consequence of model selection.

Data
To construct a measure of politicians' positions based on statements they have made, we employ automatic text analysis methods to parse a selection of speeches. Both text analysis and classification require a large text base-referred to as a "corpus"-to be effective. We use data compiled by Laurendale and Herzog (2016)  note that the Congressional Record may be amended, but amendments are minor and do not substantially alter the content of the text. 1,009 unique senator-sessions-pairs of a senator's name and a session of Congress, used to treat the same senator serving in multiple sessions as multiple individuals-are available. For some senators, other data is not available, so they are dropped from the relevant sections of the analysis where that is the case. In all cases, the sample contains at least 1,000 senator-sessions.
As discussed in Section 2, in the economics and political science literature, DWN scores, drawn from voteview.com, are the standard aggregations of Congressional roll-call votes into numeric estimates of political ideology. The two dimensions of these scores represent economic/redistributive policy and social/racial policy. For the time period we are studying, the first dimension is vastly more predictive and is thus the focus of our analysis. Since the goal is to assess voters' preferences at each election opportunity, each term a legislator served is treated as a data point. However, DWN scores are fixed for individual legislators across their careers to allow for comparison between sessions of Congress, so the same DWN scores are used for the same senator across different senator-sessions. 7 4 Methodology

Text Processing
Before attempting to classify senators, the corpus needs to be cleaned so that it is usable.
This cleaning is described in Appendix A. Briefly, all but English words are removed and stemming is applied to convert words with different conjugations to be the same word stem, e.g., "vote," "votes," "voted," would all become "vote." The standard algorithm for doing so comes from Porter (1980), used for example by both Gentzkow et al. (2016) and . The first discretionary step is choosing "relevant words." To do so, we sort words by number of appearances, deleting those with too many or too few. We use only word stems appearing over 1,000 times, or an average of one use per senator-session in our sample. These cutoffs are intended to eliminate words that may be highly predictive but only in virtue of their extremely narrow usage. Thus, the decision to exclude them is not only a technical choice but also a reflection of our intuition that voters are unable to scrutinize every single word a politician has ever said, only those that are reasonably frequent.
Histograms in Appendix A show that this cutoff does not meaningfully alter the data.

Party-Based Classification
Once the corpus has been pre-processed, speeches are assigned to senator-sessions. While a senator serves in three sessions of Congress before being reelected, congressional priorities change from session to session due to the reelection of House members, so this fine-grained analysis is preferable. We take a data-driven approach to classification and implement machine learning methods rather than developing intuitive but potentially biased approaches. Four classification methods are used: decision tree, naïve Bayes, support vector machine (SVM), and lasso-penalty regression. Details of all four methods are in Appendix B.
All classification methods used in this paper are based on 10-fold cross-validation, in which the algorithm partitions the data, uses the data minus one partition element to train the classifier, tests on the excluded partition element, and then repeats this process for all partition elements. We train each method using all ten sessions of Congress and test its performance on both the full sample and each individual session. We also train classifiers 9 distributions is not large enough to yield priors significantly different from 0 or 1, even if we fit fat-tailed distributions instead of normal distributions. A 0/1 prior renders classification moot. As such, we choose not to include a naïve Bayes analogue for this portion of the analysis, as it does not significantly affect the conclusions of this work. Finally, adapting the lasso model is straightforward. Following the observed distributions, we fit a lasso model to each party separately and assume a normally distributed outcome variable.
There remains the question of validating the accuracy of these DWN1 predictions; since the DWN1 scores are continuous, requiring that the prediction match the DWN1 score is not an option, and requiring a match within a given range leaves open the question of what level of error is natural. As such, we opt to validate in a manner that allows comparison to the results of Section 4.2. We label senators with predicted DWN1 scores of less than 0 to be Democrats and senators with predicted DWN1 scores of greater than 0 to be Republicans, then compute the accuracy rate of this assignment. Table 3 in Appendix C shows the results of this validation in a manner similar to Table 2 in the same appendix.

Discussion of Numerical Results
While all methods show the power of text for predicting party affiliation, the full-sample lasso classifier outperforms the other full-sample classification methods. The top-left entry of each matrix in Table 2 gives the accuracy rate across the full sample. The four classifiers have accuracy rates of 74.53%, 72.75%, 89.99%, and 98.32% for tree, naïve Bayes, SVM, and lasso respectively. The mean single-session classifiers have accuracy rates of 65.07%, 72.89%, 76.08%, and 67.85%, showing that the lasso does not perform as well across the full sample with limited data. However, it tends to outperform the others in sessions of Congress more distant from the one used to train the classifier. This result should be interpreted with caution. Cross-validation performed on the tree, SVM, and naïve Bayes classifiers ensures that training data is never used for testing by partitioning data into n bins and returning n sub-classifiers, each valid for 1/n of the data; in this work, we use n = 10. The overall classifier's performance is computed as the sum of correct predictions made by each subclassifier within its relevant bin for within-sample tests and the average of the sub-10 classifiers' performances for out-of-sample tests. In contrast, the lasso classifier uses crossvalidation to set the λ parameter in the lasso formula (Equation (2) in Appendix B). The algorithm then outputs one set of coefficients to be used for all data points, including those that were used as training data in some of the cross-validated attempts. Thus, a reasonable approximation for the lasso performance within-sample would be to consider by how much the lasso outperforms a rate of 0.9-equivalent to successfully fitting all the training data-and scaling that up by 10. This is a lower bound, as the lasso algorithm explicitly avoids over-fitting and thus the resulting classifier may "intentionally" misclassify some training data. Under this specification, the lasso still tends to outperform the other classifiers, albeit with the caveat that this rough approach is not apt for precise comparisons.
Across methods, there is a rough decrease in performance as the test data is drawn from sessions further away from the training data, indicating evolving trends in partisan speech.
The naïve Bayes and SVM classifiers have least variance in their accuracy rate. We can measure this by computing the standard deviation of the accuracy rate among the singlesession classifiers of each type, then taking the average of those standard deviations. The tree has a value of 0.1972 compared to 0.0848 for the SVM model and 0.0954 for naïve Bayes model. However, the naïve Bayes classifier does somewhat worse on nearby sessions than the others, perhaps indicating that the assumption of independence of each word's distribution (discussed in Equation (1) in Appendix B) is violated. Both the naïve Bayes and SVM classifiers tend to perform slightly worse in-sample than in nearby sessions. This phenomenon has a straightforward explanation. In-sample testing is done so that no partition classifies its own training data, ensuring out-of-sample behavior. In neighboring sessions, even senators used for training are considered "out-of-sample," so a partition trained on a senator in, e.g., the 105th session counts accurate classification of that senator in the 106th session.
This "quasi-in-sample" effect works in the opposite direction of the divergence of vocabulary over time. Thus, for the naïve Bayes and SVM classifiers, the quasi-in-sample effect dominates because the drop-off in performance over time is not as steep. This mathematical effect provides support for two intuitive claims: first, that senators' vocabulary is more constant during the terms they serve than the vocabulary of Congress at large, and second, 11 that over time, Congressional language changes both among and between parties.
Also of interest are occasions on which classifiers yield a result less than 0.5, indicating that predicting the opposite of what a classifier suggests would be more accurate than chance, i.e., the classifier is "backwards." While this could be a technical result related to the particularities of the classifier, it occurs across three different classification methods and thus that is unlikely to always be the case. This issue is most common for the classification tree, which relies on binary assessments. It occurs 42 times compared to 6 for the SVM, 3 for the naïve Bayes model, and 15 for the lasso model. We therefore propose a reason for this empirical observation: certain words may "change party" over time. For instance, it seems intuitive that words like "budget" or "pass" (as in, pass a bill) would be associated with the majority party, while "veto" might be associated with the minority party, especially if they hold the presidency. The coefficients of the lasso-penalty regression represent the words whose counts are most useful in classification, and thus can be interpreted as representing one measure of word partisanship. Extracting words with the greatest magnitude of change in coefficients between each pair of sessions of Congress would allow detailed analysis of how partisan vocabulary changes between sessions. This analysis is left for future work.
Classification using DWN1 scores as validated by party classification does not have a significant effect on the performance of the tree classifier. The whole-Congress tree performs 2.20% worse on average when trained with DWN1 data and the average individual-session classifier performs 0.85% worse on average. However, no classifier's average performance changes by more than 5%. No session of Congress becomes more than 2% more difficult to classify on average, though no session becomes easier to classify on average. Other than this mild overall decrease in performance, there is no clear pattern regarding the tree-session pairs that result in better or worse accuracy rates.
Comparing the party-trained SVM to the DWN1-trained SVM, the whole-Congress SVM performs markedly worse when trained with DWN1 data, with an accuracy rate about 20% lower in each Congress and overall. However, the individual-Congress SVMs show no clear change in average performance. The average change in average performance is a 0.33% decline in accuracy, with individual changes ranging from a 6.33% average improvement in 12 the 111th Congress SVM to a 5.50% decline in performance in the 112th Congress SVM.
No session of Congress becomes more than 3% easier or more difficult to classify across all individual-Congress SVMs. Also of note is the absence of a clear pattern as to which SVMsession pairs result in better or worse accuracy rates. These results point to the difficulty faced by the whole-Congress SVM being a result of the aggregate variation of DWN1 scores rather than any given Congress being difficult to classify.
These results suggest that introducing the intermediate step of determining senator ideology before assigning party classifications does not provide any clear benefit for classification, and may in fact serve as a hindrance. However, the lasso classifier shows precisely the opposite result, rising from its already-impressive average performance of around 70% accuracy to near-perfect classification when this intermediate step is introduced.
Indeed, the lasso's performance only falls below 90% in three lasso-session pairs, with the lasso generated from the 106th Congress achieving accuracy rates of 72.12%, 87.88%, and 86.00% when classifying the 108th, 109th, and 110th Congresses respectively. This markedly different result clarifies that the DWN1 scores are not uninformative, but that their informational value depends on the model through which they are interpreted. Because the lasso model achieves precise estimates of the DWN1 scores, it is then able to leverage those estimates into extremely accurate classification. The lasso model's errors in estimating DWN1 scores are on the order of 0.01 or less; this means only those scores that are already close to 0, or those senators whose scores cross the ideological midpoint, will lead to misclassification. In contrast, the larger error rates in score prediction exhibited by the tree and SVM classifiers mean that the DWN1 scores may not be more valuable than directly classification through party labels.

Conclusion
This work applies automatic text classification models to the question of party identity. In its simplest form, this is the task that voters face at the ballot box. Understanding whether and how this behavior can be replicated computationally provides insight into voters' ability to sort candidates by ideology. We show that political speech data is a powerful tool for predicting partisan affiliation, with the most accurate models consistently achieving 13 accuracy rates around or above 90% and all models consistently achieving accuracy rates above 70%. Predicting DW-NOMINATE scores instead of party affiliation and then extrapolating party affiliation from those predictions has mixed results. While the lasso model improves significantly in accuracy, the tree and SVM models perform slightly worse.
These results imply that some models may be more effective tools for this data than others, and that the choice of model ought to reflect the specific task it is intended for. Overall, all the results here show that accurate identification of party from political text is both possible and probable. The decreasing effectiveness of classification as the data to be classified moves further away in time from the training data is an important result showing that temporal variation is present in Congressional speech. This conclusion indicates that voters who do not keep up with political developments will likely lose the ability to perform the basic partisan-sorting task after only one or two sessions of Congress.
There are three primary limitations to this analysis, which point to directions for future work. The first is the classification models used, which do not cover the range of machine learning techniques. Most notably, neural network approaches, e.g., Goodfellow et al. (2017), were not considered, in large part due to the volume of data needed for training.
However, refinements of these methods, e.g., transfer learning, have the potential to further improve upon the already-successful results of the classification in this work. Secondly, there is room for improvement with respect to the datasets used. While the corpus drawn from the Congressional Record is extensive and thorough, it may not be a fully accurate representation of the public image of politicians. This provides ground for further research that uses web-scraping and other techniques to perform similar text-based analysis using public speeches and statements, providing an even more accurate picture of outward -facing political behavior. Finally, there are a variety of non-word-count-based methods of text analysis, such as word embedding or natural language processing, e.g., Mikolov et al. (2013), that could incorporate more complex relationships between words and phrases or even include meaning as a component of the analysis. These approaches would further enrich both the predictive power and the verisimilitude of these models as they relate to the task faced by voters. The realm of text analysis is a growing field with great potential for 14 applications in political science, economics, and beyond. This work is a first step towards taking full advantage of these rich new tools and datasets. What is not producing is not the industrial worker in the United States, it is the U.S. Congress. We haven't produced. We have been running around like lemmings: Free trade, free trade, fast track, fast track-having no idea in the Lord's world what we are doing; whereas we are exporting jobs faster than we can create them.

A2: Data Cleaning
Data cleaning and analysis was performed in Matlab. The key steps in cleaning (with corresponding Matlab functions in parentheses) are: erase punctuation (erasePunctuation), convert to lowercase (lower), create an array of documents (tokenizedDocument), remove common "stop words" (removeWords(stopwords)), remove words with too few or too many characters (removeShortWords, removeLongWords), apply the Porter stemmer from Section 4.1 (normalizeWords), and create a model containing word counts (bagOfWords).

Appendix B: Classification Methods
The first method used is a decision tree classifier. Decision tree classification functions by forming a "tree" of queries, called nodes, that result in binary or numeric responses.
Depending on the result of the previous query, a new query is proposed until the data has been sorted. For example, a simple classification tree might ask, "Is 'Iraq' used >100 times?" If yes, it asks, "Is 'tax' used > 65 times?" and if no, it asks "Is 'foreign' used >75 times?" Each data point-here, a word count for a given senator-travels a singular path along the branches. Node choice is determined by an algorithm based around impurity, the degree to which the branches of a node line up with the classes for classification.
The second method is a naïve Bayes classifier. This approach treats each word count as a random variable that, conditional on the speaker being Republican or Democratic, is 18 independent of other word counts. The naïve Bayes classifier estimates the distributions of these random variables and then implements Bayes' Rule: (1) where Y is party affiliation, the Xj are the random variables for word j's count, and k = 1or 2 is party. π is the prior probability of belonging to one of the classes, here the proportion of the total population in a given class. Each word used must be used by enough senators to allow distribution-fitting to take place. As such, about 50 words were dropped; however, none of those words appears in the top 1,000 most common words.
The third classifier studied here is a support vector machine, or SVM. An SVM constructs a set of hypersurfaces in high-dimensional space to separate the testing data. Then where β0 is the intercept, β is the vector of the coefficients βj, Dev represents the deviance (here to the log-likelihood of the classifier fitting the true data), and λ is the lasso penalty parameter. This penalty produces a classifier capturing the true effect of the predictor variables, not all of which are likely to be relevant. The value of Dev depends on the choice of "link function," which we choose to represent the distribution of our data.
For party classification, we assume a binomial distribution and use a logit link function 19 log[µ/(1µ)] = Xb where µ is our dependent variable, X is the matrix of word counts, and b is the vector of coefficients we are trying to find. For DWN1 classification, we assume a normal distribution and use the identity link function µ = Xb. A lasso classifier also generates, via the coefficients with largest magnitude, a set of "most partisan words." We use this list to validate that the words dropped from the naïve Bayes classification due to insufficient data are not particularly partisan.