{'fraction_non_alphanumeric': 0.029909748638770793, 'fraction_numerical': 0.0126799433131946, 'mean_word_length': 5.001790510295434, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Exploiting Class Labels to Boost Performance on Embedding-based Text Classification

Text classification is one of the most frequent tasks for processing textual data, facilitating among others research from large-scale datasets. Embeddings of different kinds have recently become the de facto standard as features used for text classification. These embeddings have the capacity to capture meanings of words inferred from occurrences in large external collections. While they are built out of external collections, they are unaware of the distributional characteristics of words in the classification dataset at hand, including most importantly the distribution of words across classes in training data. To make the most of these embeddings as features and to boost the performance of classifiers using them, we introduce a weighting scheme, Term Frequency-Category Ratio (TF-CR), which can weight high-frequency, category-exclusive words higher when computing word embeddings. Our experiments on eight datasets show the effectiveness of TF-CR, leading to improved performance scores over the well-known weighting schemes TF-IDF and KLD as well as over the absence of a weighting scheme in most cases.


INTRODUCTION
Word embeddings, or distributed word representations, have become one of the most common features for processing text. Word embeddings have been successfully used in numerous NLP and IR tasks such as sentiment analysis [3], machine translation [29], search [8] or recommender systems [18], as well as different domains such as biomedicine [4] or finance [5], outperforming traditional vector representation methods based on bags-of-words or n-grams.
In this work we focus on text classification [23], where word embeddings and derivatives are commonly used to represent the textual content of the instances to be classified [27]. While word embeddings are widely used features for text classification, they are generally used for vector representation of the textual content of the instances, independent of the importance each word can have within each category. We propose to incorporate information derived from category labels in the training data to improve vector representations. Here we propose Term Frequency-Category Ratio (TF-CR), a weighting scheme that exploits the category labels from training data to produce an improved vector representation using word embeddings, which is informed by category distributions in the training data.
There is a dearth of work in the literature trying to exploit the distribution of content across classes in the training data for improving word embedding representations. The few works that tackled the problem have mainly focused on doing so for large-scale datasets, where it is possible to train separate word embedding models for each category thanks to the availability of abundant in-domain data. This is the case for problems such as sentiment analysis, where one can build such large annotated datasets by using distant supervision. In this paper, we aim to develop an improved word embedding representation for text classification where training data is not necessarily so abundant. To do that, we are the first to propose a novel weighting scheme, Term Frequency-Category Ratio (TF-CR), which can be applied on pre-trained, domain-agnostic word embedding models, only leveraging the training data available in the dataset at hand for dataset-specific weighting of the embeddings. The intuition behind TF-CR is to assign a higher weight to high-frequency, category-exclusive words as observed in the training data.
Our experiments on eight classification datasets show the consistent effectiveness of TF-CR, significantly improving performance of word embedding representations for text classification over the use of the well-known weighting schemes TF-IDF and KLD, as well as over unweighted word embeddings.

RELATED WORK
Early methods for learning distributed representations of words [2] through the so-called neural probabilistic language models have more recently gained momentum as embeddings [11,20]. It sparked development of additional methods to reduce the dimensionality of traditional vector representation methods such as bags-of-words, by learning word embeddings. Two of the best-known methods to learn word embeddings include Word2Vec [17] and Glove [19], which enable dimensionality reduction as well as capturing semantic similarities across words. The key intuition is that, having a large corpus to train a model from, one can learn semantic characteristics of words by analysing their context, i.e. words surrounding other words. This leads to vectors of reduced dimensionality (word embeddings) to represent each word, which are normally between 100 and 500 dimensions. One of the widely adopted practices for sentence representation is then to get the sum or the average of the word embeddings in the sentence in question [28].
Use of word embeddings for text classification without specific weighting of words, however, ignores potentially useful information that can be extracted from class labels. While this problem has been tackled before, there is limited work exploring the utility of class labels to make the most of word embeddings for text classification. Previous work leveraging class labels to boost the performance of word embeddings on text classification has largely focused on sentiment analysis. The sentiment analysis task is suitable as it is possible to collect large, distantly supervised datasets [9] which are exploited to train sentiment-specific embeddings. Having large annotated datasets, one can then train separate word embedding models for each class or learn models that incorporate class distributions in them. This has been achieved in different methods by combining multiple neural networks [14,15,[24][25][26] or by using separate training processes [30] to train different word embedding models for each class in the dataset. This however requires availability of very large collections of labelled data to train separate models, which is possible for classification tasks exploiting distant supervision for data collection, as is the case with sentiment analysis. However, it is more limited for other text classification problems where gathering labelled data is expensive. In what follows, we propose a new weighting scheme to tackle this problem, TF-CR.

THE TF-CR WEIGHTING SCHEME
We propose a novel weighting scheme for word embedding representation in text classification tasks, which aims to determine the importance of each word for each particular category based on the distribution of the word across categories in the training data; this can provide additional information that word embeddings inherently disregard. This can be achieved by using well-known weighting schemes which are often used for text classification based on bags-of-words, such as TF-IDF [13,22] and Kullback-Leibler Divergence (KLD) [16].
To suit the purposes of word embeddings in text classification, here we propose a new weighting scheme. The Term Frequency-Category Ratio (TF-CR) is a simple weighting scheme that combines the importance of a word within a category (Term Frequency, TF) and the distribution of the word across all categories (Category Ratio, CR). Both TF and CR are computed for each word w within each category c. TF is computed as the ratio of words in a category that are w, i.e. T F wc = |w c | N c , where |w c | is the number of occurrences of w in c, and N c is the total number of words in c. CR is computed as the ratio of occurrences of w that occur within the category c, i.e. CR wc = |w c | |w | , where |w | denotes the number of occurrences of w across all categories.
The final TF-CR is the product of both metrics (Equation 1).
TF-CR ultimately gives a high weight to words that occur exclusively and with high frequency within a category. Low-frequency words exclusive to a category and high-frequency words that frequently occur across all categories will get lower scores.

Applying TF-CR on embeddings
In order to create a representation weighted using TF-CR, we first build category-specific word embedding representations of a text. This category-specific representation is created by summing up the embeddings of each of the words in a sentence, multiplied by their TF-CR score. This leads to k TF-CR-weighted embedding representations, where k is the number of categories in the dataset. We finally concatenate these k embedding representations to produce the final vector, which has a dimensionality of k × d, where d is the number of dimensions of the word embedding model.

EXPERIMENTS 4.1 Datasets
We use eight different datasets: • RepLab polarity dataset [1]: A dataset of 84,745 tweets mentioning companies, annotated for polarity as positive, negative or neutral. 1 • ODPtweets [31]: a large-scale dataset with nearly 25 million tweets, each categorised into one of the 17 categories of the Open Directory Project (ODP). • Restaurant reviews [12]: a large dataset of 14,542,460 Tri-pAdvisor restaurant reviews with their associated star rating ranging from 1 to 5. • SemEval sentiment tweets [21]: we aggregate all annotated tweets from the SemEval Twitter sentiment analysis task from 2013 to 2017. The resulting dataset contains 61,767 tweets. • Distantly supervised sentiment tweets: by using a large collection of tweets from January 2013 to September 2019 released on the Internet Archive 2 , we produce a dataset of tweets annotated for sentiment analysis by using distant supervision following [9], leading to tweets annotated as positive or negative.

Weighting Schemes
We compare four different weighting schemes, all of which are applied following the methodology in §3.1: • No weighting (no wgt).
• TF-IDF, which weights words with low document frequency higher. We compute TF-IDF scores for each word within each category, therefore calculating the importance of the word in each category. • KLD, which determines the saliency of a word in a category with respect to the rest of the categories. Again KLD leads to a score for each word in each category. • TF-CR, our weighting scheme defined in §3.

Varying Sizes of Training Sets
While we have up to 90,000 instances available as training data, we perform experiments with varying numbers of training instances. This allows us to assess the extent to which weighting schemes can help with varying sizes of training data, provided that calculations of weights using these schemes are done solely from the training data available in each case. We performed experiments for training sets ranging from 1,000 to 9,000. Training instances are randomly sampled in each training scenario, keeping the random sample consistent across different experiments with the same training size, and incrementally adding instances, i.e. a training set of 5,000 contains all of the training instances of that with 4,000 plus another 1,000. All performance scores reported are the result of averaging 10-fold cross-validation experiments. Table 1 shows the results with varying numbers of training instances. We observe that TF-CR consistently outperforms the other weighting schemes, TF-IDF and KLD, regardless of the training size. The gap between TF-CR and the other weighting schemes generally becomes larger as the training data increases, showing that TF-CR exploits the class distributions more effectively. We can also observe that the unweighted approach outperforms TF-CR in six out of eight datasets when the training data is as small as 1,000 instances. However, TF-CR becomes more effective as the training data increases. TF-CR outperforms the unweighted method in five out of eight datasets with 10,000 training instances, and in seven out of eight datasets with 90,000 training instances. This reinforces the effectiveness of TF-CR for mid-sized training sets and above. Figure 1 shows the tendency of all four methods as the training size varies from 1,000 to 90,000, with steps of 1,000. With the exception of the hate speech dataset, TF-CR outperforms all other methods for larger training sets. Moreover, TF-CR consistently outperforms all other methods for most training sizes in five datasets: 20newsgroups, newsspace200, odptweets, restaurants and sentiment.

DISCUSSION
We have introduced TF-CR, 7 a first-of-its-kind weighting scheme that can leverage word distributions across categories in training data for text classification. The intuition behind TF-CR is to give higher weights to frequent words that exclusively or predominantly occur within a category. This leads to category-specific weights for each word, which allows an embedding representation that captures varying importances of words across categories. Experimenting on eight datasets, we show that (1) it improves over unweighted word embeddings in seven of the datasets with large training datasets, 7    (2) it improves consistently for most training sizes in five of the datasets. TF-CR also consistently outperforms TF-IDF and KLD.
Our objective here has been to introduce and validate TF-CR. Additional tuning of classifier parameters, adding features, etc. for achieving state-of-the-art performance is beyond the scope of this work. We also aim to extend this work by further exploring the differences across datasets, to determine dataset characteristics that maximise the benefits of TF-CR.