{'fraction_non_alphanumeric': 0.03505213001839883, 'fraction_numerical': 0.01698353540595367, 'mean_word_length': 4.760326086956522, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Text Classification Improved by Integrating Bidirectional LSTM with Two-dimensional Max Pooling

Recurrent Neural Network (RNN) is one of the most popular architectures used in Natural Language Processsing (NLP) tasks because its recurrent structure is very suitable to process variable-length text. RNN can utilize distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. And this matrix includes two dimensions: the time-step dimension and the feature vector dimension. Then most existing models usually utilize one-dimensional (1D) max pooling operation or attention-based operation only on the time-step dimension to obtain a fixed-length vector. However, the features on the feature vector dimension are not mutually independent, and simply applying 1D pooling operation over the time-step dimension independently may destroy the structure of the feature representation. On the other hand, applying two-dimensional (2D) pooling operation over the two dimensions may sample more meaningful features for sequence modeling tasks. To integrate the features on both dimensions of the matrix, this paper explores applying 2D max pooling operation to obtain a fixed-length representation of the text. This paper also utilizes 2D convolution to sample more meaningful information of the matrix. Experiments are conducted on six text classification tasks, including sentiment analysis, question classification, subjectivity classification and newsgroup classification. Compared with the state-of-the-art models, the proposed models achieve excellent performance on 4 out of 6 tasks. Specifically, one of the proposed models achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.


Introduction
Text classification is an essential component in many NLP applications, such as sentiment analysis (Socher et al., 2013), relation extraction (Zeng et al., 2014) and spam detection (Wang, 2010). Therefore, it has attracted considerable attention from many researchers, and various types of models have been proposed. As a traditional method, the bag-of-words (BoW) model treats texts as unordered sets of words (Wang and Manning, 2012). In this way, however, it fails to encode word order and syntactic feature.
Recently, order-sensitive models based on neural networks have achieved tremendous success for text classification, and shown more significant progress compared with BoW models. The challenge for textual modeling is how to capture features for different text units, such as phrases, sentences and documents. Benefiting from its recurrent structure, RNN, as an alternative type of neural networks, is very suitable to process the variable-length text.
RNN can capitalize on distributed representations of words by first converting the tokens comprising each text into vectors, which form a matrix. This matrix includes two dimensions: the time-step dimension and the feature vector dimension, and it will be updated in the process of learning feature representation. Then RNN utilizes 1D max pooling operation (Lai et al., 2015) or attention-based operation (Zhou et al., 2016), which extracts maximum values or generates a weighted representation over the time-step dimension of the matrix, to obtain a fixed-length vector. Both of the two operators ignore features on the feature vector dimension, which maybe important for sentence representation, therefore the use of 1D max pooling and attention-based operators may pose a serious limitation.
Convolutional Neural Networks (CNN) (Kalchbrenner et al., 2014;Kim, 2014) utilizes 1D convolution to perform the feature mapping, and then applies 1D max pooling operation over the time-step dimension to obtain a fixed-length output. However the elements in the matrix learned by RNN are not independent, as RNN reads a sentence word by word, one can effectively treat the matrix as an 'image'. Unlike in NLP, CNN in image processing tasks (LeCun et al., 1998; applies 2D convolution and 2D pooling operation to get a representation of the input. It is a good choice to utilize 2D convolution and 2D pooling to sample more meaningful features on both the time-step dimension and the feature vector dimension for text classification. Above all, this paper proposes Bidirectional Long Short-Term Memory Networks with Two-Dimensional Max Pooling (BLSTM-2DPooling) to capture features on both the time-step dimension and the feature vector dimension. It first utilizes Bidirectional Long Short-Term Memory Networks (BLSTM) to transform the text into vectors. And then 2D max pooling operation is utilized to obtain a fixed-length vector. This paper also applies 2D convolution (BLSTM-2DCNN) to capture more meaningful features to represent the input text.
The contributions of this paper can be summarized as follows: • This paper proposes a combined framework, which utilizes BLSTM to capture long-term sentence dependencies, and extracts features by 2D convolution and 2D max pooling operation for sequence modeling tasks. To the best of our knowledge, this work is the first example of using 2D convolution and 2D max pooling operation in NLP tasks.
• This work introduces two combined models BLSTM-2DPooling and BLSTM-2DCNN, and verifies them on six text classification tasks, including sentiment analysis, question classification, subjectivity classification, and newsgroups classification. Compared with the state-of-the-art models, BLSTM-2DCNN achieves excellent performance on 4 out of 6 tasks. Specifically, it achieves highest accuracy on Stanford Sentiment Treebank binary classification and fine-grained classification tasks.
• To better understand the effect of 2D convolution and 2D max pooling operation, this paper conducts experiments on Stanford Sentiment Treebank fine-grained task. It first depicts the performance of the proposed models on different length of sentences, and then conducts a sensitivity analysis of 2D filter and max pooling size.
The remainder of the paper is organized as follows. In Section 2, the related work about text classification is reviewed. Section 3 presents the BLSTM-2DCNN architectures for text classification in detail. Section 4 describes details about the setup of the experiments. Section 5 presents the experimental results. The conclusion is drawn in the section 6.

Related Work
Deep learning based neural network models have achieved great improvement on text classification tasks. These models generally consist of a projection layer that maps words of text to vectors. And then combine the vectors with different neural networks to make a fixed-length representation. According to the structure, they may divide into four categories: Recursive Neural Networks (RecNN 1 ), RNN, CNN and other neural networks.
Recursive Neural Networks: RecNN is defined over recursive tree structures. In the type of recursive models, information from the leaf nodes of a tree and its internal nodes are combined in a bottom-up manner. Socher et al. (2013) introduced recursive neural tensor network to build representations of phrases and sentences by combining neighbour constituents based on the parsing tree. Irsoy and Cardie (2014) proposed deep recursive neural network, which is constructed by stacking multiple recursive layers on top of each other, to modeling sentence.
Recurrent Neural Networks: RNN has obtained much attention because of their superior ability to preserve sequence information over time. Tang et al. (2015) developed target dependent Long Short-Term Memory Networks (LSTM (Hochreiter and Schmidhuber, 1997)), where target information is automatically taken into account. Tai et al. (2015) generalized LSTM to Tree-LSTM where each LSTM unit gains information from its children units. Zhou et al. (2016) introduced BLSTM with attention mechanism to automatically select features that have a decisive effect on classification. Yang et al. (2016) introduced a hierarchical network with two levels of attention mechanisms, which are word attention and sentence attention, for document classification. This paper also implements an attention-based model BLSTM-Att like the model in Zhou et al. (2016).
Convolution Neural Networks: CNN (LeCun et al., 1998) is a feedforward neural network with 2D convolution layers and 2D pooling layers, originally developed for image processing. Then CNN is applied to NLP tasks, such as sentence classification (Kalchbrenner et al., 2014;Kim, 2014), and relation classification (Zeng et al., 2014). The difference is that the common CNN in NLP tasks is made up of 1D convolution layers and 1D pooling layers. Kim (2014) defined a CNN architecture with two channels. Kalchbrenner et al. (2014) proposed a dynamic k-max pooling mechanism for sentence modeling. (Zhang and Wallace, 2015) conducted a sensitivity analysis of one-layer CNN to explore the effect of architecture components on model performance. Yin and Schütze (2016) introduced multichannel embeddings and unsupervised pretraining to improve classification accuracy. (Zhang and Wallace, 2015) conducted a sensitivity analysis of one-layer CNN to explore the effect of architecture components on model performance.
Usually there is a misunderstanding that 1D convolutional filter in NLP tasks has one dimension. Actually it has two dimensions (k, d), where k, d ∈ R. As d is equal to the word embeddings size d w , the window slides only on the time-step dimension, so the convolution is usually called 1D convolution. While d in this paper varies from 2 to d w , to avoid confusion with common CNN, the convolution in this work is named as 2D convolution. The details will be described in Section 3.2.
Other Neural Networks: In addition to the models described above, lots of other neural networks have been proposed for text classification. Iyyer et al. (2015) introduced a deep averaging network, which fed an unweighted average of word embeddings through multiple hidden layers before classification. Zhou et al. (2015) used CNN to extract a sequence of higher-level phrase representations, then the representations were fed into a LSTM to obtain the sentence representation.
The proposed model BLSTM-2DCNN is most relevant to DSCNN  and RCNN (Wen et al., 2016). The difference is that the former two utilize LSTM, bidirectional RNN respectively, while this work applies BLSTM, to capture long-term sentence dependencies. After that the former two both apply 1D convolution and 1D max pooling operation, while this paper uses 2D convolution and 2D max pooling operation, to obtain the whole sentence representation.

Model
As shown in Figure 1, the overall model consists of four parts: BLSTM Layer, Two-dimensional Convolution Layer, Two dimensional max pooling Layer, and Output Layer. The details of different components are described in the following sections.

BLSTM Layer
LSTM was firstly proposed by Hochreiter and Schmidhuber (1997) to overcome the gradient vanishing problem of RNN. The main idea is to introduce an adaptive gating mechanism, which decides the degree to keep the previous state and memorize the extracted features of the current data input. Given a sequence S = {x 1 , x 2 , . . . , x l }, where l is the length of input text, LSTM processes it word by word. At time-step t, the memory c t and the hidden state h t are updated with the following equations: (1) where x t is the input at the current time-step, i, f and o is the input gate activation, forget gate activation and output gate activation respectively,ĉ is the current cell state, σ denotes the logistic sigmoid function and ⊙ denotes element-wise multiplication. For the sequence modeling tasks, it is beneficial to have access to the past context as well as the future context. Schuster and Paliwal (1997) proposed BLSTM to extend the unidirectional LSTM by introducing a second hidden layer, where the hidden to hidden connections flow in opposite temporal order. Therefore, the model is able to exploit information from both the past and the future.
In this paper, BLSTM is utilized to capture the past and the future information. As shown in Figure  1, the network contains two sub-networks for the forward and backward sequence context respectively. The output of the i th word is shown in the following equation: Here, element-wise sum is used to combine the forward and backward pass outputs.

Convolutional Neural Networks
Since BLSTM has access to the future context as well as the past context, h i is related to all the other words in the text. One can effectively treat the matrix, which consists of feature vectors, as an 'image', so 2D convolution and 2D max pooling operation can be utilized to capture more meaningful information.

Two-dimensional Convolution Layer
A matrix H = {h 1 , h 2 , . . . , h l }, H ∈ R l×d w , is obtained from BLSTM Layer, where d w is the size of word embeddings. Then narrow convolution is utilized (Kalchbrenner et al., 2014) to extract local features over H. A convolution operation involves a 2D filter m ∈ R k×d , which is applied to a window of k words and d feature vectors. For example, a feature o i,j is generated from a window of vectors H i:i+k−1, j:j+d−1 by where i ranges from 1 to (l − k + 1), j ranges from 1 to (d w − d + 1), · represents dot product, b ∈ R is a bias and an f is a non-linear function such as the hyperbolic tangent. This filter is applied to each possible window of the matrix H to produce a feature map O: with O ∈ R (l−k+1)×(d w −d+1) . It has described the process of one convolution filter. The convolution layer may have multiple filters for the same size filter to learn complementary features, or multiple kinds of filter with different size.

Output Layer
For text classification, the output h * of 2D Max Pooling Layer is the whole representation of the input text S. And then it is passed to a softmax classifier layer to predict the semantic relation labelŷ from a discrete set of classes Y . The classifier takes the hidden state h * as input: A reasonable training objective to be minimized is the categorical cross-entropy loss. The loss is calculated as a regularized sum: where t ∈ R m is the one-hot represented ground truth, y ∈ R m is the estimated probability for each class by softmax, m is the number of target classes, and λ is an L2 regularization hyper-parameter. Training is done through stochastic gradient descent over shuffled mini-batches with the AdaDelta (Zeiler, 2012) update rule. Training details are further introduced in Section 4.3. The proposed models are tested on six datasets. Summary statistics of the datasets are in Table 1.
• MR 2 : Sentence polarity dataset from Pang and Lee (2005). The task is to detect positive/negative reviews.
• SST-1 3 : Stanford Sentiment Treebank is an extension of MR from Socher et al. (2013). The aim is to classify a review as fine-grained labels (very negative, negative, neutral, positive, very positive).
• SST-2: Same as SST-1 but with neutral reviews removed and binary labels (negative, positive). For both experiments, phrases and sentences are used to train the model, but only sentences are scored at test time (Socher et al., 2013;Le and Mikolov, 2014). Thus the training set is an order of magnitude larger than listed in table 1.
• Subj 4 : Subjectivity dataset (Pang and Lee, 2004). The task is to classify a sentence as being subjective or objective.
• 20Newsgroups 6 : The 20Ng dataset contains messages from twenty newsgroups. We use the bydate version preprocessed by Cachopo (2007). We select four major categories (comp, politics, rec and religion) followed by Hingmire et al. (2013).

Word Embeddings
The word embeddings are pre-trained on much larger unannotated corpora to achieve better generalization given limited amount of training data (Turian et al., 2010).

Hyper-parameter Settings
For datasets without a standard development set we randomly select 10% of the training data as the development set. The evaluation metric of the 20Ng is the Macro-F1 measure followed by the state-ofthe-art work and the other five datasets use accuracy as the metric. The final hyper-parameters are as follows.
The dimension of word embeddings is 300, the hidden units of LSTM is 300. We use 100 convolutional filters each for window sizes of (3,3), 2D pooling size of (2,2). We set the mini-batch size as 10 and the learning rate of AdaDelta as the default value 1.0. For regularization, we employ Dropout operation  with dropout rate of 0.5 for the word embeddings, 0.2 for the BLSTM layer and 0.4 for the penultimate layer, we also use l2 penalty with coefficient 10 −5 over the parameters.
These values are chosen via a grid search on the SST-1 development set. We only tune these hyperparameters, and more finer tuning, such as using different numbers of hidden units of LSTM layer, or using wide convolution (Kalchbrenner et al., 2014), may further improve the performance.

Overall Performance
This work implements four models, BLSTM, BLSTM-Att, BLSTM-2DPooling, and BLSTM-2DCNN. Table 2 presents the performance of the four models and other state-of-the-art models on six classification tasks. The BLSTM-2DCNN model achieves excellent performance on 4 out of 6 tasks. Especially, it achieves 52.4% and 89.5% test accuracies on SST-1 and SST-2 respectively. BLSTM-2DPooling performs worse than the state-of-the-art models. While we expect performance gains through the use of 2D convolution, we are surprised at the magnitude of the gains. BLSTM-CNN beats all baselines on SST-1, SST-2, and TREC datasets. As for Subj and MR datasets, BLSTM-2DCNN gets a second higher accuracies. Some of the previous techniques only work on sentences, but not paragraphs/documents with several sentences. Our question becomes whether it is possible to use our models for datasets that have a substantial number of words, such as 20Ng and where the content consists of many different topics. For that purpose, this paper tests the four models on document-level dataset 20Ng, by treating the document as a long sentence. Compared with RCNN (Lai et al., 2015), BLSTM-2DCNN achieves a comparable result.
Besides, this paper also compares with ReNN, RNN, CNN and other neural networks: • Compared with ReNN, the proposed two models do not depend on external language-specific features such as dependency parse trees.
• CNN extracts features from word embeddings of the input text, while BLSTM-2DPooling and BLSTM-2DCNN captures features from the output of BLSTM layer, which has already extracted features from the original input text.
• BLSTM-2DCNN is an extension of BLSTM-2DPooling, and the results show that BLSTM-2DCNN can capture more dependencies in text.
• AdaSent utilizes a more complicated model to form a hierarchy of representations, and it outperforms BLSTM-2DCNN on Subj and MR datasets. Compared with DSCNN , BLSTM-2DCNN outperforms it on five datasets.

Effect of Sentence Length
It is found that both BLSTM-2DPooling and BLSTM-2DCNN outperform the other two models. This suggests that both 2D convolution and 2D max pooling operation are able to encode semantically-useful structural information. At the same time, it shows that the accuracies decline with the length of sentences increasing. In future work, we would like to investigate neural mechanisms to preserve long-term dependencies of text.

Effect of 2D Convolutional Filter and 2D Max Pooling Size
We are interested in what is the best 2D filter and max pooling size to get better performance. We conduct experiments on SST-1 dataset with BLSTM-2DCNN and set the number of feature maps to 100.
To make it simple, we set these two dimensions to the same values, thus both the filter and the pooling are square matrices. For the horizontal axis, c means 2D convolutional filter size, and the five different color bar charts on each c represent different 2D max pooling size from 2 to 6. Figure 3 shows that different size of filter and pooling can get different accuracies. The best accuracy is 52.6 with 2D filter size (5,5) and 2D max pooling size (5,5), this shows that finer tuning can further improve the performance reported here. And if a larger filter is used, the convolution can detector more features, and the performance may be improved, too. However, the networks will take up more storage space, and consume more time.

Conclusion
This paper introduces two combination models, one is BLSTM-2DPooling, the other is BLSTM-2DCNN, which can be seen as an extension of BLSTM-2DPooling. Both models can hold not only the time-step dimension but also the feature vector dimension information. The experiments are conducted on six text classificaion tasks. The experiments results demonstrate that BLSTM-2DCNN not only outperforms RecNN, RNN and CNN models, but also works better than the BLSTM-2DPooling and DSCNN . Especially, BLSTM-2DCNN achieves highest accuracy on SST-1 and SST-2 datasets. To better understand the effective of the proposed two models, this work also conducts a sensitivity analysis on SST-1 dataset. It is found that large filter can detector more features, and this may lead to performance improvement.