{'fraction_non_alphanumeric': 0.02639416658108247, 'fraction_numerical': 0.0070863715723528805, 'mean_word_length': 4.93026796589525, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Incorporating Effective Global Information via Adaptive Gate Attention for Text Classification

The dominant text classification studies focus on training classifiers using textual instances only or introducing external knowledge (e.g., hand-craft features and domain expert knowledge). In contrast, some corpus-level statistical features, like word frequency and distribution, are not well exploited. Our work shows that such simple statistical information can enhance classification performance both efficiently and significantly compared with several baseline models. In this paper, we propose a classifier with gate mechanism named Adaptive Gate Attention model with Global Information (AGA+GI), in which the adaptive gate mechanism incorporates global statistical features into latent semantic features and the attention layer captures dependency relationship within the sentence. To alleviate the overfitting issue, we propose a novel Leaky Dropout mechanism to improve generalization ability and performance stability. Our experiments show that the proposed method can achieve better accuracy than CNN-based and RNN-based approaches without global information on several benchmarks.


Introduction
Text classification is playing an essential role in Natural Language Processing (NLP) as one of the fundamental tasks with broad applications. The mainstream deep text classifiers suffer from the data sparseness issue, and to enrich semantic features, researchers turn to some useful external knowledge as complementary information, i.e., tags, character, POS, sentiment lexicon, entity knowledge base. Their studies show that introducing proper external knowledge is helpful to the classification task. However, we notice that some most primitive features are overlooked in the deep learning era, i.e. word frequency and distribution, which are fixed and intrinsic feature of a corpus. The most representative algorithm utilizing statistical feature is the term frequency-inverse document fre- * Contact Author quency (TFIDF), which is a straightforward information retrieval technique for document modelling. However, because of the bag-of-word nature, TFIDF is unable to utilize positional information and capture fine-grained semantics, which makes it a less favourable feature compared with word embeddings in the deep architecture. Nevertheless, to our surprise, we find that using term-count-of-labels statistics (defined in Section 2.1) as an auxiliary feature shows substantial improvements in the pilot study, in which the word frequency adapts weights of terms via a simple attention layer. We believe that the researchers may underestimate the real power of global statistics feature in deep learning; therefore, in this work, we design a new framework to fuse such features elegantly.
When designing the fusion mechanism, we think of two major concerns: 1. The semantic feature and statistical feature are not compatible in scale and dimension; 2. The new information may be not necessary for all semantic features. To address the first concern, we employ non-linear projection to map both features into a shared information space to make both latent representations compatible with each other. The second concern raises a new perspective towards the method of using additional information 1 . We argue that not every semantic feature need to be enhanced since some additional information may introduce noise to the classifier. Therefore, instead of element-wise operation, we design an Adaptive Gate module to add auxiliary information to the less-confident semantic features only (with values around 0.5 after sigmoid activation) while the high-confident semantic features remain unchanged. By doing this, the proposed model can achieve a balance between the original semantic features and the additional features for better decision making.
Moreover, we notice that the current Dropout mechanism may not be compatible with the proposed feature fusion mechanism. Neurons enriched by additional information and neurons containing preserved semantic features may be deactivated during training, and the improvement brought by the new architecture can be offset. Therefore, instead of directly deactivating neurons, we propose a novel Leaky Dropout mechanism to reduce the value of selected neurons only and conduct further experiments to demonstrate the effectiveness of this novel mechanism.
The main contributions of this paper are summarized as follows: • We leverage corpus-level statistics feature to enrich semantic features for text classification. To retrieve necessary and useful global information only, we propose a well-designed adaptive gate module with attention mechanism to fuse statistics feature into semantic features with low confidence. • We propose a novel Leaky Dropout to alleviate overfitting issue without fully deactivating neurons, which is demonstrated to be more robust than conventional Dropout. • We conduct extensive experiments on six small-scale datasets and two large-scale datasets with significance test. The results show that our models significantly outperform state-of-the-art methods.

Global information
We first formally define the adopted global information as follows.
Definition 1 Term-count-of-labels (TCoL) is a global statistics of a term towards the labels. Given a word w and a set of labels of c classes, the TCoL vector w is where ζ i is the count of word w on label i. Given a sentence The term-count-of-labels describes the global distribution of labels as a feature of the word. Such features are primitive but highly informative in feature selection. Intuitively, if a word has very high or very low frequency on all labels, then we shall assume this word has limited contribution to the classification task. In contrast, if a word appears more frequently in one class, we assume this word is more informative and hope our classifier can highlight this word in decision making. The TCoL dictionary is obtained only from the training set. Figure 1 shows a generic framework of the proposed model, consisting of input layer, feature-extraction layer, adaptive gate attention (AGA) module and output layer. Since there exists different feature-extraction techniques, we implement AGA-CNN and AGA-LSTM as two variants of the model employing CNN and LSTM as feature extractor respectively. Given an annotated documents, the AGA+GI model firstly maps the document into embedding vector matrix and deploy convolutional or recursive operation to obtain latent semantic representations. Then, a well-designed AG module projects semantic representation and TCoL matrix to shared information space. It fuses global statistic features into the pipeline by selectively combining units from both matrices, in which the global information makes semantic representation more informative. Finally, the model extracts higher-level features utilizing attention layers and forwards the features in fullyconnected layers to output logits for label prediction and loss calculation. During the training, we compute cross-entropy loss and employ ADAM optimizer to train the classifier.

Input layer
The input of model is a sentence with fixed length m and the TCoL matrix ζ of the sentence. we first map each word into a k-dimensional continuous space and obtain the word embedding vector x m ∈ R k . Then we concatenate all word vectors to form a k × m matrix as model input: We pad the sentences to keep a uniform length for all sentences following the same way in [Kim, 2014].

Feature extraction layer
We employ convolutional operation or recurrent operation to produce latent feature map. For a CNN layer, we apply a filter W f ∈ R h×k with window size h. The new feature c i is generated from a window of word vectors x i:i−h+1 : here, b ∈ R is the bias term, and f (·) is a non-linear function. Each filter produces a feature vector c = [c 1 , c 2 , . . . , c m ] with padding. We employ d filters in this layer to produce a latent feature map C ∈ R d×m in the semantic space.
For an LSTM layer, we follow the same settings with [Hochreiter and Schmidhuber, 1997] and [Zhou et al., 2015]. At time t, the latent feature c t in each LSTM cell is obtained as follow, c t = LSTM(c t−1 , x t ).
(5) With hidden dimension set to d, we have the whole semantic feature C ∈ R d×m .

Adaptive Gate Attention module
To merge global information with semantic features, we map the semantic feature map C and the normalized TCoL matrix ζ s to a shared information space H ∈ R d×m through the fully-connected layers as follow, where H C and H ζ are latent representations of semantic feature and term frequency respectively in the information space, and ζ s is normalized by dividing the length of the dictionary V . We find this projection is very crucial in Section 3.4. The AG module fuses H C and H ζ to output a GI-enhanced feature map H O through the AdaGate function, where ReLU(·) and σ(·) are activation functions, and stands for element-wise production. The σ(·) function produces values in probability form, and the Valve function is designed to restore less-confident entries (with probability near 0.5) for matching with elements in H ζ . More concretely, for every unit a ∈ H C , where is a leaky hyper-parameter tuning the threshold of confidence, specifically, we dump all statistical information if = 0, and combine statistical information with all semantic features if = 0.5. Therefore, the element-wise production exploits Valve(σ(H C ), ) as a filter to extract necessary global information only, thus H O can be more informative by restoring both essential semantic features and additional GI features. After fusing GI, we normalize H O via softmax to get attention weight α, Then we apply attention weight to the semantic representation C to produce feature vector a,

Output layer & loss function
After passing through fully-connected layers with Leaky Dropout (discussed in Section 2.3) and softmax layer, feature vector a is mapped to the label space for label prediction and loss calculation. To maximize the probability of the correct label y, we deploy optimizer to minimize cross-entropy loss L, which is defined as

Leaky Dropout
The Dropout [Srivastava et al., 2014] is an elegant mechanism to alleviate overfitting issue when training a deeper network as shown in Figure 2a, but directly blocking a selected neuron may have some unexpected side effects. Therefore, we propose a soft mechanism called Leaky Dropout to suppress the weights of selected neurons rather than completely deactivate them, as shown in Figure 2b. Given an input feature vector x and dropout rate β, the partially suppressed vector x is obtained by computing element-wise production of x and a mask vector m: and the value of each element in m is assigned as: where z i is sampled from a Bernoulli distribution as indicators (preserve if z = 1 and suppress if z = 0), and parameter c is set to control how hard the suppression will be. Following the same setting in [Srivastava et al., 2014], we magnify preserved cells to keep expectations unchanged. 3 Experiment

Datasets
We test the proposed model on various datasets. CR [Hu and Liu, 2004] Table 1. We deploy 10-fold cross-validation in the datasets without standard train/test split. Due to the limitation of computation power, we randomly sample 1 million data from Yelp F. and Yelp P. and divide sampled data into train/test sets.

Baselines
We compare the proposed model with following text classifiers. To evaluate the contribution of frequency information explicitly, we use TFIDF as features and apply SVM as classifier on small datasets. TextCNN [Kim, 2014] et al., 2017] treats the average of word/ngrams embeddings as document embeddings, then feeds doc-ument embeddings into a linear classifier. AGA-CNN w/o GI and AGA-LSTM w/o GI are baselines for ablation study with set to 0, which means the gate rejects all additional information, but we preserve the projection C → H C .

Implementation details Evaluation metrics
We evaluate the model performance and the significance of improvement using the following metrics. F1 score measures both precision and recall as a whole. We report Macroaverage results on both multi-class and binary-class datasets in this paper. Accuracy measures how many instances are correctly classified among all instances. T-test reveals how significant the improvements are and we report p-value of the proposed model compared with baselines for each trail.

Word embedding
It is a widely adopted approach to improve model performance by initializing word vectors with pre-trained language model. We adopt the publicly available FastText 2 [Mikolov et al., 2018], which has 1 million word vectors trained on Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (16B tokens) with the dimensionality of 300. Words not present in the pre-trained model are initialized randomly.

Parameter settings
The parameters involved in all CNN and RNN modules follow the settings in their original papers. More concretely, the CNN-based models have filter size of [3,4,5] with 100 filters of each, and the RNN-based models have hidden dimension of 128. All models adopt Adam optimizer with batch size of 64 and dropout rate of 0.5 .

Results & Discussion
Results of our models against other methods and results of ablation study are listed in Table 2 (multi-class datasets) and Table 3 (Binary-class datasets) respectively. In general, the proposed model achieves the best accuracy on all datasets and best F1 score on most datasets (except TREC). The t-test indicates the proposed methods have significant improvement in the majority of all results (50 out of 63). We conduct additional experiments to validate the effectiveness of the proposed Leaky Dropout and show the results in Figure 3.

Effect of AGA module
We had initially designed the projection layer and the attention layer of AGA module as a designated mechanism to incorporate GI. However, as shown in Table 2 and 3, the AGA module can improve the performance of CNN-based model significantly on all datasets even without GI. Specifically, we see improvements of 5.23% on F1 of SST-1, 2.17% on F1 of Yelp F. and 1.78% on accuracy of Yelp P., suggesting the proposed method is more potent on semantic feature selection than the original max-over-time pooling method. Besides, compared with all baseline models, models with AGA module have achieved significant improvements on dataset Yelp F. and Yelp P., which indicates that AGA module may be more suitable to deal with complicated dependency relationship when the data amount is enormous.   Table 3: Results on Binary datasets. We evaluate the performance using Accuracy (also equal to F1 score). Baseline results with p-value indicated means our methods have significant improvement compared with this baseline. is the leaky parameter in the adaptive gate.

Effect of GI
The contribution of GI is distinctly identified as models with GI achieve the best performance on most datasets in the ablation experiments. We also remark that the improvements brought forth by GI can be affected by the leaky constant in the gate module, which controls the confidence interval to trigger the information fusion. On large datasets, a larger can produce higher accuracy than a smaller , while on small datasets, a smaller is more favourable (0.05 in this case), which possibly due to the bias from TCoL information on small datasets. As defined previously, TCoL is a frequency statistic of words towards labels, which can be easily deviated from the real distribution by the limited size of a dataset.
To deal with such a problem, a tighter interval is preferred in order to make the model depend more semantic features, which can alleviate the influence of the bias. In contrast, for large datasets, the TCoL statistics can approximate the real distribution and be more helpful to the model training.

Effect of Leaky Dropout
To compare the vanilla Dropout with Leaky Dropout, we apply both mechanisms with AGA-CNN, AGA-LSTM, TextCNN and BiLSTM on TREC dataset and visualize the curves of Accuracy and Loss during the training in Figure 3. As shown in the figure, the Leaky Dropout generally produces a better performance when c = 500. Specifically, the models with Leaky Dropout achieve a higher accuracy and show better convergence than the models with vanilla Dropout and without Dropout on the test set, which means the generalization ability of the model improves. Furthermore, the tail sections of curves reveal that Leaky Dropout is more robust to overfitting issue compared with conventional Dropout as there is severe fluctuation on curves of both accuracy and loss for models with conventional Dropout. Also, as shown in Fig achieve a higher accuracy that those with the conventional Dropout, which naturally makes sense since leaky mechanism can preserve infused information while the conventional Dropout may drop the neurons enriched by GI information.
4 Related Work

Text classification
Existing approaches employ deep architecture for supervised text classification and can achieve remarkable performance. Kim [Kim, 2014] proposes a classic TextCNN model for text classification, which significantly improved the accuracy of the classification task compared with machine learning approach.  [Yang et al., 2016] propose the Hierarchical Attention Network to imitate the hierarchical structure of sentences and capture both word-and sentencelevel features. These works mainly focus on architecture design for better feature extraction and selection, while our work coalesces semantic features with additional information and highlights the design of the fusion mechanism.

Classifier with additional knowledge
There is a large body of relevant literature to enhance classification performance using external knowledge in NLP. Researchers create and exploit many active features incorporating information from various domains, including but not limited to linguistics, psychology and knowledge base. [ Wang et al., 2017] conceptualize sentence as a set of concepts using taxonomy knowledge base and obtain the embeddings by merging concepts on top of pre-trained word vectors, which can capture ampler contextual information facilitated by deep models. These works fail to concern the necessity and compatibility of added information, which probably bring more noise to the original semantic features and increase the cost of computation.

Conclusion & Future work
In this paper, we propose the Adaptive Gate Attention module to incorporate global statistical features and conduct extensive experiments with both CNN-based framework and RNNbased framework to show the effectiveness of the proposed method. The proposed AGA module can merge necessary global information only while preserving essential semantic features, in which we provide a deep insight into the framework design of introducing additional knowledge. Moreover, the AGA module has great flexibility in use and can be extended to various relevant works. We also propose a novel Leaky Dropout mechanism to enhance the model generalization ability to enhance the model generalization ability and conduct additional experiments to demonstrate its effectiveness. Due to the page limit, we cannot do a comprehensive review with complete experiments on the Leaky Dropout, so we plan to examine the Leaky Dropout with theoretical analysis in our future work.