{'fraction_non_alphanumeric': 0.02399823866138265, 'fraction_numerical': 0.0100543079407016, 'mean_word_length': 4.920703888768195, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Actionable and Political Text Classification using Word Embeddings and LSTM

In this work, we apply word embeddings and neural networks with Long Short-Term Memory (LSTM) to text classification problems, where the classification criteria are decided by the context of the application. We examine two applications in particular. The first is that of Actionability, where we build models to classify social media messages from customers of service providers as Actionable or Non-Actionable. We build models for over 30 different languages for actionability, and most of the models achieve accuracy around 85%, with some reaching over 90% accuracy. We also show that using LSTM neural networks with word embeddings vastly outperform traditional techniques. Second, we explore classification of messages with respect to political leaning, where social media messages are classified as Democratic or Republican. The model is able to classify messages with a high accuracy of 87.57%. As part of our experiments, we vary different hyperparameters of the neural networks, and report the effect of such variation on the accuracy. These actionability models have been deployed to production and help company agents provide customer support by prioritizing which messages to respond to. The model for political leaning has been opened and made available for wider use.


INTRODUCTION
A large body of work in recent years in the field of text classification has explored sentiment mining [19], [18], [6], [23], [1]. In such problems, phrases or sentences are classified as positive or negative based on the sentiment expressed. While knowing the sentiment of a message certainly has its uses, other forms of useful text classification have remained relatively unexplored. In this work explore scenarios where knowing only the sentiment of a piece of text is not useful enough to get good insights. We instead consider contextual classification of text, where the classification is based on criteria determined by the application.
One such application of contextual text classification is in the area of customer support. Public interactions on social networks provide an ideal platform for customers to interact with brands, companies and service providers. According to Okeleke [16] 50% of users prefer reaching service providers on social media over contacting a call center. A user may seek customer support from a service provider on social media with a complaint or a call for assistance. In most cases where a customer may reach out in this manner, the message with the complaint expresses a negative sentiment. A service provider seeking to resolve such customer complaints may find little value in identifying that the sentiment of such messages is negative.
Instead, a better contextual classification is to classify the messages as Actionable or Non-Actionable. For such conversations, social media messages that include a clear call to action, or raise a specific issue can be categorized as Actionable. Alternatively, agents of the service provider may not be able to respond to Non-Actionable messages that are too broad, general or not related to any specific issue. The service providers could then prioritize responding to Actionable messages over non-actionable ones, saving money and resources in the process. The ability to sift through interactions and classify them as actionable can help reduce company response latency and improve efficiency, thereby leading to better customer service. A good solution to identify actionable messages can lead to large cost savings for companies by reducing call center loads.
Another example where such contextual classification becomes important, is with respect to voter opinions during elections. During the election year in the United States, people on social media platforms such as Twitter may talk about various political issues, some leaning left with Democratic views and some leaning right with Republican views. While expressing such views, a particular individual may speak positively about certain issues and negatively about others. In this scenario, merely understanding the sentiment of messages is not sufficient to provide enough insight into voter opinions. Instead messages classified as Democratic or Republican may provide greater insight into the political preferences of different groups of people, and such classification may prove to be more valuable to a political candidate trying to understand his or her voters.
In recent studies, neural networks have shown great promise in performing sentiment and text classification tasks [19], [18], [11]. Further, word embeddings have proven to be use-ful semantic feature extractors [13], [17], and long-short term memory (LSTM) networks [7] have been shown to be quite effective in tasks involving text sequences [22]. Here we apply word embeddings and LSTM networks to the problem of contextual text classification, and present experiments and results for the two applications described above.
Our contributions in this study are as follows: • Actionability: We build models to classify messages as actionable or non-actionable, to help company agents providing customer support by prioritizing which messages to respond to. These models have been deployed to production.
• Political Leaning: We build a second set of models to predict political leaning from social media messages, and classify them as Democrat or Republican. We open this model and make it available for wider use.
• RNN experiments: We vary different hyperparameters of the RNN models used for classifying text in the applications above, and provide a report of the effect of each hyperparameter on the accuracy.
• Multi-Lingual models: We build models for over 30 different languages for actionability using the LSTM neural networks, and also compare these results with traditional machine learning techniques.

RELATED WORK
Deep learning techniques have shown remarkable success in text processing problems. The authors in [11], [5] and [24] have applied Convolutional Neural Networks (CNNs) for a variety of NLP and text processing tasks with great results. Recurrent Neural Networks (RNNs) have also shown to be very effective for text classification [19], and LSTM networks [7] in particular have shown to perform well for sequence based learning tasks [22]. Furthermore, word embeddings such as those in word2vec [13] or GloVe [17], map words and phrases to a lower dimensional space, which serve as semantic feature extractors that can be effectively used for training. Given the inherently sequential nature of messages and sentences, we employ LSTM units in our neural networks in this work, with word embeddings for semantic feature extraction.
A significant body of work regarding the text classification has focused on the sentiment analysis. A wide range of research has been carried out, including the use of partof-speech tagging [1], lexicon approaches [23], rule-based [4], hybrid [2], [3], and deep learning [6], [19], [18]. A particular challenge for supervised learning for sentiment classification is that of human disagreements on labels, limiting models to have precision and recall values in the range of 0.75 − 0.82 [15]. While the majority of research investigates text classification problems within a unified domain, it has been recognized that sentiment detection models may change depending on the domain context [10], [6]. Here we further explore this idea of contextual text classification, and investigate classification of messages on social media based on criteria other than sentiment.
The first problem we consider is that of actionability of messages with respect to customer support on social media platforms. Munro [14] studied this problem of actionability in the context of the disaster relief, where subword and spatiotemporal models were used to classify short-form messages. Jin et. al. [9] used KNN and SVM techniques to identify customer complaints on a dataset containing 5, 500 messages from an online Chinese hotel community. Although the study showed promising results the dataset was limited and constrained to a very specific domain. Our work here builds on our previous work in [20], where manually extracted features and traditional supervised learning techniques were applied to this problem.
Another problem we consider is that of identifying political leaning of messages, by analyzing social media posts by users of such platforms. A similar study was carried out by the authors in [8], where the datasets considered were derived from debate transcripts and articles by authors with well-known political leanings. Here we instead analyze this problem for a dataset derived from social media messages, which are far more noisy in nature, and include colloquial language, abbreviations and slang. As noted by the authors in [8], for politics in the United States, a voter from either party may have a mixture of conservative or liberal views. While the work in [8] classified documents as liberal or conservative, here we classify messages along party lines, as Democratic or Republican.

METHODOLOGY
Below, we walk through the pre-processing steps and the neural network architecture used for building our models.

Pre-processing
The input sample messages are tokenized, converting the message to a sequence of tokens which serves as the input to the neural network. The tokens are also used to create a vocabulary of words to be used during training. The frequency of each token is used as the index representation for the token. The vocabulary to be used is then chosen as the top V most frequent words seen. This vocabulary size also represents the maximum features considered in the input layer of the neural network.
Each message is thus transformed into a sequence of indices corresponding to the tokens in the message. For example a phrase "the school" may be transformed to [1,125]. If an out-of-vocabulary (OOV) token is encountered, then it is marked with a reserved index.
We choose to use fixed length vectors during the training process. We therefore select a sufficiently large length as the maximum permissible length for a message, and pad zero index values to a message when it is shorter than the maximum length.

Neural Network Architecture
Deep neural networks have shown success in a variety of machine learning tasks such as image recognition, speech processing and natural language processing. By using multiple processing layers with non-linear transformations, deep neural networks are able to model high-level abstractions in data. Recurrent Neural Networks (RNNs) are a class of neural networks that are able to model relationships in time. Unlike traditional neural networks, an RNN uses units with internal states that can persist information about previous events. This makes RNNs suitable for problems that require sequential information, such as text processing tasks.
Traditional RNNs suffer from the problem of not being able to learn long-term dependencies in data due to the prob-  [7]. Such long term dependencies become an important consideration when learning to classify messages.
The neural network used here employs word embeddings with LSTM units to perform contextual text classification. We feed the pre-processed input to the neural network, and use the labels to perform supervised learning on the messages. Figure 1 shows these layers pictorially, and we describe each layer below.

Embedding Layer
The first layer in the network is an embedding layer that turns positive integer indices in the input into dense real valued vectors of fixed size, determined by the number of units in this layer. As a toy example, an input such as [[0], [4], [20]] may transformed to [[0.0, 0.0], [0.25, 0.1], [0.6, -0.2]]. The purpose of the embedding layer is to learn a mapping that embeds each word in the discrete vocabulary to a lower dimensional continuous vector space. Such a distributed representation of vocabulary words has been shown to have great advantages in extracting relationships between word concepts in [13] and [17]. The use of this layer enables semantic feature extraction from the input, without manual definition of features such as those in [1] and [20]. The output of this layer is fed as the input to the further layers in the network.

LSTM Layer
Here we use a layer with multiple LSTM units as the second layer in the network. An LSTM unit is a memory cell composed of four main components: an input gate, a selfrecurrent connection, a forget gate and an output gate. The input gate either blocks or allows an incoming signal to alter the cell-state. Similarly, the output gate either prevents or allows the cell-state from having han effect on other units. The forget gate allows the cell to remember or forget its previous state by controlling the cell's self-recurrent connection. An LSTM layer with multiple units can be thought of as a deep network across time steps, where each time step represents a layer.
In later sections we describe the effects of varying the number of units in the Embedding and LSTM layers.

Dropout Layer
Dropout [21] is a regularization technique used in neural networks to avoid overfitting. This is achieved by randomly dropping a fraction of the units while training a neural network, thus preventing units from co-adapting. The outputs of the LSTM layer in our network are fed to a Dropout layer where half the units dropout.

Fully Connected Layer
The fully connected layer has full connections to all the activations in the previous layer. This layer is essentially used to learn the non-linear combinations of the high-level features learnt by the previous layers in the network.

Loss Layer
The final layer in the network is the Loss layer that determines how deviation of the predicted labels from the actual labels are penalized. Since we are interested in binary classification of messages, here we use binary crossentropy as the loss function.
Using this framework, we train our models for the two different applications of actionability and political leaning and evaluate the results in the following sections.

ACTIONABILITY
The first set of experiments is performed for the problem of actionability, where messages are classified as actionable or non-actionable.

Datasets
The dataset for this problem is derived from a social media management platform that helps agents respond to customer posts on behalf of their company. The platform prioritizes incoming messages from customers and routes them to the appropriate agents, and is integrated with major social networks such as Twitter, Facebook and Google+, as well as with brand communities and online forums. This tool that is used by a number of brands and companies for customer support.
The labels for supervised training are gathered as follows. The company agents using the platform may respond to certain incoming messages, and ignore those that they cannot provide a good response to. If an agent provided a response, then the message is marked as Actionable (labeled 1), otherwise it is marked as Non-Actionable (labeled 0). This labeled dataset is then used for training and evaluating models.
In this study we used a trailing window of 6 months of data, from November 1st 2014 to May 1st 2015 1 . Equal number of actionable and non-actionable messages are sampled to create each dataset. The training and test datasets are created as 80/20 splits of the sampled data.
We examine actionability across languages in our experiments. A pair of training and test datasets is therefore created for each language, as well as one dataset that includes all languages. Some languages have sparse data and smaller number of samples, while others are larger. Our dataset sizes vary from 27k messages for the language with the least data (Farsi), to 8.5 million messages for the language with the most (English). A combined language dataset is also prepared, which contains 1.7 million sampled messages across all languages. Table 1 shows the training and test dataset sizes for each language.

Multi-lingual contexts
Messages in different languages add an additional dimension to the problem of actionability. We therefore generate separate models for over 30 different languages and compare their performance.
Languages such as Czech (cs), Finnish (fi), Slovak (si) and Swedish (sv) show the highest accuracies of over 90%. The lowest accuracies are for languages such as Arabic (ar), Farsi (fa) and Thai (th), where the accuracies are slightly below 80%, though in some cases like fa and th, the low accuracy may be attributed to sparsity of data. As expected, certain languages are more predictable in terms of actionability than others, and we observe a variation of 15% between the least and most accurate models. Encouragingly, we find from Table 1 that most languages show high accuracies of greater than 85%.
In addition, we also build a combined model trained on 1.7 million messages containing sampled data from all languages. Interestingly, this general model applicable across languages is able to capture actionability with a relatively high accuracy of 86%. This suggests that the neural network can capture features across languages to identify actionability.

Comparison with Traditional Methods
In [20] we explored this problem through the lens of traditional supervised learning techniques, including logistic regression, AROW, perceptrons and Adagrad RDA. Here we compare the results obtained using these traditional techniques, against the ones obtained using the LSTM network. In addition, in the previous study we extract features that are manually defined, whereas here we use word embeddings to extract semantic features.
We find from Table 2 that the LSTM models consistently outperform the best traditional supervised models for almost all the languages, with only Arabic showing a slightly worse performance. Further the improvement is significantly large with the LSTM model, the largest improvement being 15% for Italian (it). This clearly demonstrates the superiority of neural networks when it comes to the text classification problems such as actionability.

Vocabulary size
One of the parameters we consider while building models is the vocabulary size V . The vocabulary used for training the models in each language is chosen by picking the top V most frequent words seen in the corpus. This parameter also determines the dimension of the feature space used for classification. A higher dimension could presumably capture more information, at the cost of larger models and slower training times.
In our experiments we consider two vocabulary sizes for each language. The first is a smaller vocabulary of size 20, 000, and the second is a larger one of size 100, 000. The larger vocabulary adds significant overhead in training time, since more params need to be learnt by the model. Table 1 shows the comparison of accuracies for the two vocabulary sizes.
In general, we observe that there is little difference between the test accuracy values for the two vocabulary sizes, with no decisive winner among them. Thus even a smaller vocabulary is able to effectively capture actionability in messages, with the added advantage of faster training times.

Examples
A useful side effect of the predictions made with the model, is that along with the final classification, the prediction score for a given message is indicative of how strongly actionable a message may be. Table 3 shows several examples of messages, ranging from least actionable messages with low scores, to most actionable ones with scores close to 1.0. We find that messages that simply convey some news are classified as the least actionable, and the ones which state an explicit issue with a call to action are classified as highly actionable. Broad complaints or general questions are classified as non-actionable, or weakly actionable, which is a highly desired aspect when sifting through such messages.
These examples prove that the models are highly effective in classifying messages based on actionability. Also note that there are many examples where a user may show a strong negative sentiment in the message, without the message being very actionable. This again, highlights the need to classify messages on criteria other than sentiment alone.

POLITICAL LEANING
The second text classification problem we address is that of identifying political leaning with respect to politics in the United States. In this problem, we would like to classify messages as Democratic or Republican, based on the views expressed in the message.

Dataset
For this problem, we build the training and test datasets as follows: First we pick users on Twitter whose political leanings are known to be either Democratic or Republican. We do this by using Twitter Lists, which are manually curated topical lists of users by other users of the social media platform. For the users on these lists we sample the messages that they posted over a period of 3 months, between Oct 12th, 2015 to Jan 12th, 2016. The messages posted by the known Democrats as per the Twitter Lists are labeled 0, and those by known Republicans are labeled 1. We then divide this dataset into the training and test sets with an 80/20 percent split. The training set contains 336, 000 messages and the test set contains 84, 000 messages.

Training and Evaluation
For training the model, we use a similar architecture as the one used for the actionability problem above, with 128 units each in the embedding and LSTM layers. The model achieves a 88.82% accuracy on the training set, and a 87.57% accuracy on the test set.
We examine the effects of changing various parameters on the accuracy of the model. When the units in the Embedding layer are lower (32) or higher (512), the accuracy is also higher, but if the number of units is too low (16) the accuracy drops. Figure 2a shows this graphically.
• LSTM Layer Units: When the Embedding layer is held fixed at 128 units, the highest accuracy is seen when the LSTM layer has 32 units. Increasing the number of units beyond 32 lowers the accuracy, and remains more or less constant thereafter. Figure 2b shows the change in accuracy as a function of the LSTM layer units.
• Embedding and LSTM Layer Units: When the Embedding and LSTM layers have the same number of units, larger sizes show higher accuracies, but the difference is nevertheless small. Figure 2c shows a distinct increasing trend in accuracy as the number of units are increased.
• Batch size: A smaller batch size gives slightly higher accuracy, but the convergence is faster for larger batch sizes. Based on our experiments, we pick a batch size of 64, which gives a relatively high accuracy of 87.57% with a relatively fast training speed.
• Activation: Finally, we observed the performance when the activation function in the last layer was changed. The sigmoid activation performed better than the tanh activation, with a difference of 0.5% in the accuracy. Figure 2 suggests that there may be an optimal number of Embedding and LSTM layer units that leads to the highest accuracy. While we do see an increasing trend in accuracy when the Embedding and LSTM layers are simultaneously increased, but understanding how such hyperparameters should be chosen remains an open problem. This line of inquiry requires further investigation into the factors that determine the optimal number of units in the Embedding and LSTM layers for a known input size. We note that since the changes in accuracy are relatively small overall, using a smaller network has advantages in terms of training times and memory constraints.
The model built with 128 units in the embedding and 32 units in the LSTM layer, trained with sigmoid activation and the Adam optimizer is available online for download 2 . The list of users whose messages were aggregated, along with their known political leaning has also been made available.

Examples
As with actionability, the predictions for political leaning are accompanied by the prediction score, which indicate the extent of the leaning of the message. Thus a message with a score of 0.9 may express a strongly Republican view, whereas  that with a score of 0.55 may still be Republican, but may convey a more moderate view. Table 4 shows several examples of predictions made by the model. We clearly see that the model is able to classify extreme views with very high or very low scores, and moderate views with scores closer to 0.5. Again, observe that Democratic as well as Republican messages both show positive and negative sentiments, irrespective of their leaning. Further, different users affiliated to the same party may have differing sentiments regarding certain issues. This is another scenario where sentiment classification by itself may not provide enough insight into voter opinions, whereas classifying messages as Democratic or Republican may prove to be much more valuable.

CONCLUSION
In this study, we explore two useful applications of text classification that go beyond sentiment analysis. For the purpose of such classification, we employ neural networks with word embedding and LSTM layers, and examine various parameters that affect the model.
The first application we consider is that in the context of customer support on social media, where company agents respond to customer complaints. For this problem, we classify social media messages from customers as actionable or non-actionable, so that company agents can prioritize which messages to respond to. We break down this problem and solve it for each language separately, and find that the models perform very well for most languages, with over 90% accuracy for some of them. We also find that the LSTM networks outperform traditional learning techniques by a large margin. Further a relatively small vocabulary size of 20, 000 and a training set of size 330, 000 is able to capture the features required to predict actionability.
The second application we investigate is that of political leaning, where we classify messages as Democratic or Republican. We try variations of different parameters of the neural networks and observe their effects on the accuracy. Overall we see an accuracy of around 87% in the best performing models for political leaning, and provide several illustrative examples of their effectiveness.
In the case of actionability most customer complaints carry a negative sentiment, whereas for political leaning both sides display a mixture of positive and negative sentiments. As such, sentiment proves to be less valuable for such applications, since it is not a strong indicator of actionability or political leaning. We therefore make the case here, that going beyond sentiment analysis to classify text based on other contextual criteria can lead to large benefits in various applications.
The actionability models described here are deployed in production and are used by paying customers, and the model for classifying political leaning has been made open and available for wider use.

ACKNOWLEDGMENT
We would like to thank colleague David Gardner who was instrumental in deploying system actionability scoring service in production.