{'fraction_non_alphanumeric': 0.035184588102531035, 'fraction_numerical': 0.015718200870546508, 'mean_word_length': 4.877072477498816, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Japanese Sentiment Classification with Stacked Denoising Auto-Encoder using Distributed Word Representation

Traditional sentiment classification methods often require polarity dictionaries or crafted features to utilize machine learning. However, those approaches incur high costs in the making of dictionaries and/or features, which hinder generalization of tasks. Examples of these approaches include an approach that uses a polarity dictionary that cannot handle unknown or newly invented words and another approach that uses a complex model with 13 types of feature templates. We propose a novel high performance sentiment classification method with stacked denoising auto-encoders that uses distributed word representation instead of building dictionaries or utilizing engineering features. The results of experiments conducted indicate that our model achieves state-of-the-art performance in Japanese sentiment classification tasks.


Introduction
As the popularity of social media continues to rise, serious attention is being given to review information nowadays. Reviews with positive/negative ratings, in particular, help (potential) customers with product comparisons and to make purchasing decisions. Consequently, automatic classification of the polarities (such as positive and negative) of reviews is extremely important.
Traditional approaches to sentiment analysis utilize polarity dictionaries or classification rules. Although these approaches are fairly accurate, they depend on languages that may require significant amounts of manual labor. Further, dictionary-based methods have difficulty dealing with new or unknown words.
Machine learning-based methods are widely adopted in sentiment classification in order to mitigate the problems associated with the making of dictionaries and/or rules. One of the most basic features used in machine learning-based sentiment classification is the bag-of-words feature (Wang and Manning, 2012;Pang et al., 2002). In machine learningbased frameworks, the weights of words are automatically learned from a training corpus instead of being manually assigned.
However, the bag-of-words feature cannot take syntactic structures into account. This leads to mistakes such as "a great design but inconvenient" and "inconvenient but a great design" being deemed to have the same meaning, even though their nuances are different; the former is somewhat negative whereas the latter is slightly positive. To solve this syntactic problem, Nakagawa et al. (2010) proposed a sentiment analysis model that used dependency trees with polarities assigned to their subtrees. However, their proposed model requires specialized knowledge to design complicated feature templates.
In this study, we propose an approach that uses distributed word representation to overcome the first problem and deep neural networks to alleviate the second problem. The former is an unsupervised method capable of representing a word s meaning without using hand-tagged resources such as a polarity dictionary. In addition, it is robust to the data sparseness problem. The latter is a highly expressive model that does not utilize complex engineering features or models.
Our research makes the following two main contributions: • We show that distributed word representation learned from a large-scale corpus and multiple layers (more than three layers) contributes significantly to classification accuracy in sentiment classification tasks.
• We achieve state-of-the-art performance in Japanese sentiment classification tasks without designing complex features and models.

Related Works
In this section, we discuss related works from two areas: sentiment classification and deep learning (distributed word representation and multi-layer neural networks).

Sentiment classification
Sentiment classification has been researched extensively in the past decade. Most of the previous approaches in this area rely on either timeconsuming hand-tagged dictionaries or knowledgeintensive complex models. Ikeda et al. (2008) proposed a method that classifies polarities by learning them within a window around a word. Their proposed method works well with words registered in a dictionary. However, building a polarity dictionary is expensive and their approach is not able to cope with unknown words. In contrast, our proposed approach does not use a polarity dictionary and works robustly even when there are infrequent words in the test data.
In a similar manner, Choi et al. (2008) proposed a method in which rules are manually built up and polarities are classified considering dependency structures. However, the rules are based on English, which cannot be applied directly to other languages. This is unlike our method, which does not employ any language-specific rules. Nakagawa et al. (2010) proposed a supervised model that uses a dependency tree with polarity assigned to each subtree as hidden variables. The proposed approach further classifies sentiment polarities in English and Japanese sentences with Conditional Random Field (CRF), considering the interactions between the hidden variables. The dependency information enables them to take syntactic structures into account in order to model polarity flip. However, their proposed method is so complex that it has to create multiple feature templates. In contrast, our model is quite simple and does not require the engineering of such features.

Deep learning
One of the great advantages of deep learning is that it reduces the need to hand-design features. Instead, it automatically extracts hierarchical features and enhances the end-to-end classification performance learned through backpropagation. As a consequence, it avoids the engineering of task-specific ad-hoc features using copious amounts of prior knowledge. Further, it sometimes surpasses humanlevel performance (He et al., 2015). Two of the most actively studied areas in deep learning for NLP applications are representation learning and deep neural networks.
Representation learning Several studies have attempted to model natural language texts using deep architectures. Distributed word representations, or word embeddings, represent words as vectors. Distributed representations of word vectors are not sparse but dense vectors that can express the meaning of words. Sentiment classification tasks are significantly influenced by the data sparseness problem. As a result, distributed word representation is more suitable than traditional 1-of-K representation, which only treats words as symbols.
In our proposed method, to learn the word embeddings, we employ a state-of-the-art word embedding technique called word2vec (Mikolov et al., 2013b;Mikolov et al., 2013a), which we discuss in Section 3.1. Although several word embedding techniques currently exist (Collobert and Weston, 2008;Pennington et al., 2014), word2vec is one of the most computationally efficient and is considered to be state-of-the-art. Collobert et al. (2008) presented a model that learns word embedding by jointly performing multi-task learning using a deep convolutional architecture. Their method is considered to be state-of-the-art as well, but it is not readily applicable to Japanese.
Multi-layer neural networks A stacked denoising auto-encoder (SdA) is a deep neural network that extends a stacked auto-encoder (Bengio et al., 2007) with denoising auto-encoders (dA). Stacking multiple layers and introducing noise to the input layer adds high generalization ability to auto-encoders. This method is used in speech recognition (Dahl et al., 2011), image processing (Xie et al., 2012) and domain adaptation ; further, it exhibits high representation ability. Glorot et al. (2011) used SdAs to perform domain adaptation in sentiment analysis. After learning sentiment classification in four domains of the reviews of products on Amazon, they tested each model with different domains. Although the task and method are similar to those of our proposed approach, they only use the most frequent verbs as input.
Dos Santos et al. (2014) and Tang et al. (2014) researched sentiment classification of microblogs such as Twitter using the distributed representation learned by the methods of Collobert et al. (2008) and Mikolov et al. (2013b;2013a). Those two tasks are the same task as ours, but the former generats sentence vectors using string-based convolution networks while the latter utilizes a model that treats the distributed word representation itself as polarities. Our proposed approach makes sentence vectors by simply averaging the distributed word representation, yet achieves state-of-the-art performance in Japanese sentiment classification tasks. Kim (2014) classified the polarities of sentences using convolutional neural networks. He built a simple CNN with one layer of convolution, whereas our model uses multiple hidden layers. Socher et al. (2011;2013) placed common autoencoders recursively (recursive neural networks) and concatenated input vectors to take syntactic information such as the order of words into account. In addition, they arranged auto-encoders (AEs) to syntactic trees to represent the polarities of each phrase. Recursive neural networks construct sentence vectors differently from our approach. Compared to their model, our distributed sentence representation is quite simple yet effective for Japanese sentiment classification.

Sentiment Classification with Stacked
Denoising Auto-Encoder using Distributed Word Representation In this study, we treated the task of classifying the polarity of a sentence as a binary classification. Our proposed approach makes a sentence vector from the input sentence, and then inputs the sen-tence vector to a classifier. The sentence vector is computed from the average of word vectors in the sentence, based on distributed word representation.
In Section 3.1 we introduce distributed representation of words and sentences, and in Section 3.2 we explain multi-layer neural networks.

Distributed representation
1-of-K representation is a traditional word vector representation for making bag-of-words. The dimension of a word vector in 1-of-K is the same as the size of the vocabulary, and the elements of a dimension correspond to words. 1-of-K treats different words as discrete symbols. However, 1-of-K representation fails to model the shared meanings of words. For example, the word vectors "dog" and "cat" should share "animal" or "pet" meanings to a certain degree, but 1-of-K representation is not able to capture this similarity. Consequently, we propose distributed word representation.
The task of learning distributed representation is called representation learning and has been of significant interest in the NLP literature in the last few years. Distributed word representation learns a lowdimension dense vector for a word from a largescale text corpus to capture the word's features from its context.

Distributed word representation
Let the number of vocabularies be |V |, the dimension of a vector representing words be d, 1-of-K vector be b ∈ R |V | and the matrix of all word vectors be L ∈ R d×|V | . The kth target word vector w k is consequently represented as in Equation 1.
Continuous Bag-of-Words (CBOW) and Skipgram models in word2vec (Mikolov et al., 2013b;Mikolov et al., 2013a) have attracted tremendous attention as a result of their effectiveness and efficiency. The former is a model that predicts the target word using contexts around the word, whereas the latter is a model that predicts the surrounding context from the target word. According to Mikolov's work, skip-gram shows higher accuracy than CBOW 1 . Therefore, we used skip-gram in our experiments.

Distributed sentence representation
In our approach, we construct a sentence matrix S ∈ R |M |×d from the corpus containing |M | sentences.
First, we describe how to create a sentence vector from word vectors. The ith (1 ≤ i ≤ M ) input sentence composed of |N (i) | words is used to make a sentence vector S (i) ∈ R d with the word vectors.
The jth (1 ≤ j ≤ d) element of sentence vector S (i) is calculated by averaging the corresponding element of the word vectors in the sentence as expressed in Equation 2 (Figure 1).

S
Finally, the sentence matrix S is defined by Equation 3.

Auto-Encoder
An auto-encoder is an unsupervised learning method devised by Hinton and Salakhutdinov (2006) that uses neural networks. It learns shared features of the input at the hidden layer. By restricting the dimension of the hidden layer to be smaller than that of an input layer, it reduces the dimension of the input layer. The encode function that calculates a hidden layer from an input is shown in Equation 4, and the formed it. Therefore, we present only the experiments conducted using skip-gram in this paper. decode function that calculates an output layer from the hidden layer is shown in Equation 5 below.
s( * ) represents nonlinear functions such as tanh or sigmoid, W , W ′ are weight matrices and b, b ′ are bias terms, respectively. The parameters of auto-encoders are learned by minimizing the following loss functions. The loss function measures the difference between input vector x and output vector z using the cross entropy (Equation 6). We use Stochastic Gradient Descent (SGD) to minimize the loss function.

Denoising Auto-Encoder
Regularization is usually used in the loss function in traditional multi-layer perceptrons. Denoising techniques play the same role as regularization in auto-encoders.
A denoising auto-encoder is a stochastic extension of a regular auto-encoder that adds noise randomly to the input during training to obtain higher generalization ability. Because the loss function of denoising auto-encoders evaluates the input without adding noise, denoising auto-encoders can be expected to extract better representations than autoencoders (Vincent et al., 2008). DropOut (Hinton et al., 2012) achieves similar regularization objectives by ignoring the hidden nodes, not input, with a uniform probability.

Stacked Denoising Auto-Encoder
A stacked denoising auto-encoder piles dAs into multiple layers and improves representation ability. The deeper the layers go, the more abstract features will be extracted (Vincent et al., 2010). The training procedure used for SdAs comprises two steps. Initially, dAs are used to pre-train each layer via unsupervised learning, after which the entire neural network is fine-tuned via supervised learning. In the pre-training phase, feature extraction is carried out by the dAs from input A i , and the extracted hidden representation is treated as the input to the next hidden layer. After the final pre-training process, the last hidden layer is classified with softmax and the resulting vector is passed to the output layer. The fine-tuning phase backpropagates supervision to each layer to update weight matrices ( Figure 2).
In Figure 2, the input vector is obtained from Equation 2 and dA1 is applied with the weight matrix of the first layer W 1 to calculate the first hidden layer. Note that the numbers of hidden layers and hidden nodes are hyperparameters. We define n i to be the number of hidden nodes of the ith layer. Therefore, using Equation 4 the dimension of weight matrix W 1 will be n 1 × d. Similarly, the weight matrices up to the l − 1th layer will be W i ∈ R n i ×n i−1 (i > 2). At the final lth layer, we need to convert the dimension of the hidden layer into d label , the dimension of the label, so the dimension of weight matrix W l should become d label × n l−1 .

Methods
To demonstrate the effectiveness of a nonlinear SdA, we compared it with a linear classifier (logistic regression, LogRes-w2v). 2 In addition, to investigate the usefulness of distributed word representation, we compared methods using bag-of-features (LogRes-BoF, SdA-BoF). We constructed sentence vectors S ∈ R |V | with 1-of-K representation in the same manner as Equation 2, and performed dimension 2 Both SdA and logistic regression were implemented using Theano version 0.6.0. reduction to d = 200 using Principal Component Analysis (PCA). 3 We introduce a weak baseline (most frequent sense) and a strong baseline (state-of-the-art). The latter is a method by Nakagawa et al. (2010), which uses the same corpus.
MFS. The most frequent sense baseline. It always selects the most frequent choice (in this case, negative).
Tree-CRF. The state-of-the-art baseline with hidden variables learned by tree-structured CRF (Nakagawa et al., 2010).

LogRes-BoF. Performs sentiment classification using bag-of-features with a linear classifier (logistic regression).
SdA-BoF. Classifies polarity with the same input vectors as LogRes-BoF.
LogRes-w2v. Classifies polarity with a linear classifier (logistic regression) using the sentence vector computed by distributed word representation.
SdA-w2v. Our proposed method that classifies polarity with a SdA using the same input as LogRes-w2v.
We adjusted the noise rate, the numbers of hidden layers and hidden nodes, as follows.
To demonstrate the denoising efficiency, we varied the noise rate (0%, 10%, 20%, 30%, 40% and 50%) for SdAs. We then performed denoising by zeroing a vector with binomial distribution at a specified rate.
To show the effect of stacking, we increased the number of hidden layers (from 1 to 6).
To examine the representation ability of the network, we varied the number of hidden nodes (100, 300, 500, and 700).

Corpus and tools
We obtained distributed word representations using word2vec 4 with Skip-gram (Mikolov et al., 2013b;Mikolov et al., 2013a). We used Japanese Wikipedia's dump data (2014.11) to learn the 200 dimension distributed representation with word2vec after word-segmentation with MeCab 5 . The vocabulary of the models contains 426,782 words (without processing negation) and 431,782 words (with processing negation).
The corpus used in the experiment was the Japanese section of NTCIR-6 OPINION (Seki et al., 2007). The data used in our research were the sentences from The Mainichi Newspaper and The Japan News articles with polarities annotated by three annotators. For each sentence, we took the union of the annotations of the three annotators. When the annotations were split to both positive and negative, we always used the annotation of the specific annotator. The resulting corpus contained 2,599 sentences. The positive instances comprised 765 sentences whereas the negative instances comprised 1,830 sentences. Although a neutral polarity existed, we ignored it because our task is binary classification.
We performed 10-fold cross validation with 10 threads of parallel processing and evaluated the performance of binary classification with accuracy.

Results
First, Figure 3 shows the accuracy and standard errors of each method for the NTCIR-6 corpus.
It can be clearly seen that our method is superior 4 https://code.google.com/p/word2vec/ 5 MeCab version-0.996 IPADic version-2.7.0 Note that the parameters of the SdAs above are the best combination of noise rate, number of hidden layers, and number of hidden nodes (noise rate: 10%, four layers, and 500 dimensions). 6 Table 1 contrasts the various hyperparameters. We changed one parameter at a time, while leaving all other parameters fixed. The upper row compares the accuracy of the system with changing noise rate. The best result was obtained when the noise rate was set to 50%. Compared with the standard stacked auto-encoder (noise rate: 0%, accuracy: 81.1%), an SdA with a noise rate of 50% exhibits better accuracy (81.6%). In the middle of the table, we changed the number of hidden layers. It turned out that, the classifier worked best with four layers. As can be seen, the stacked auto-encoder is superior to the unstacked one by 1.0 accuracy point. At the bottom of the table, we changed the dimension of hidden nodes. We changed hidden nodes in intervals of 200 dimensions, but the accuracy only fluctuated by ±0.1 point. The accuracy was highest when the dimension was 500.

Discussion
In this section, we discuss the results of the models (Figure 3), parameter tuning (Table 1), and examples (Table 2).

Methods
BoF vs. Distributed word representation. When the model was fixed to a linear classifier (logistic regression), the accuracies with Bagof-Features and distributed word representation were 70.8% and 79.5%, respectively. In contrast, using an SdA, the result for Bagof-Features was 76.9% and that of distributed word representation was 81.7%. Considering these outcomes, it can be seen that a 4.8 to 8.7 point increase in accuracy occurred when distributed word representation was used. Hence, the contribution of distributed word representation is the largest among the different experimental settings.
Linear classifier vs. SdA. The accuracies of logistic regression and SdAs with the same word vectors made from Bag-of-Features were 70.8% and 76.9%, respectively. With distributed word representation, the accuracy of the linear classifier was 79.6% and that of SdA was 81.7%. Thus, a 2.2 to 6.1 point improvement was obtained using SdAs over a traditional linear classifier.
Negation handling. As can be seen in Figure 3, the accuracy of SdA-w2v-neg decreased by 0.8 point compared with SdA-w2v. This differs from Nakagawa et al. (2010)'s report. The reason for this phenomenon may be the data sparseness problem caused by the negation process. We checked the number of negations in the corpus and found that the numbers of types and tokens are 326 (3.8%) and 1,239 (1.0%), respectively. Thus, the negation process may have little influence on the accuracy.

Parameters
Figures 4 and 5 show the total training time obtained with 10 parallel processes by changing the numbers of hidden layers and hidden nodes. Figure 4 shows that the training time grew gradually as the number of hidden layers increased. In contrast, Figure 5 shows that the training time doubled when the number of hidden nodes was increased by 200. These results originate from the structure of SdAs. The nodes of the two adjacent hidden layers are fully connected. Hence, if the network has l layers and n dimensional nodes, the number of connections will be l × n × n = ln 2 . That indicates the relationship between the number of layers and connections is linear, but the number of connections grows exponentially with the number of nodes. Consequently, a small increase in the number of nodes results in a long training time. In contrast, as can be seen from Table 1, the number of nodes has little or no effect on accuracy, whereas changing the number of layers helps to improve the performance.

Examples
Several examples are presented in Table 2. The values P and N represent the prediction of positive and negative, respectively.
Looking at the top of the correct answer, it can be seen that our model classified polarity robustly against the data sparseness problem, such as with the coined word " (Fujimorism)" with which the BoF model is weak. Further, linear classifiers and the unstacked AE fail to handle double negative sentences such as at the bottom. Regardless of the difficulties, our model copes well with the situation.
Moving on to the wrong answers, it can be seen that our proposed model made human-like mistakes. For example, it mistook the top one containing the word " (thinking over, reflection, regret)," but it is an ambiguous sentence that might be labeled as positive. Similarly, it failed to classify the middle sentence containing the phrase " (prevent to replay the tragedy)," which ends with " (criticize)." The annotations of the above two examples were divided into both positive and negative 7 . At the bottom, the proposed method did not successfully identify the polarity flipping with the phrase " (not yield to the pressure)." Because the model with negation handling answered it correctly, there remains much room for improvement on how to deal with interactions between syntax and semantics (Tai et al., 2015;Socher et al., 2013).

Conclusion
In this study, we presented a high performance Japanese sentiment classification method that uses distributed word representation learned from a largescale corpus with word2vec and a stacked denoising auto-encoder. The proposed method requires no dictionaries, complex models, or the engineering of numerous features. Consequently, it can easily be adapted to other tasks and domains without the need for advanced knowledge from experts. In addition, due to the nature of learning with vectors, our system does not depend on languages.
As our future works, we will try to create the distributed sentence representation using the Recurrent Neural Networks (Irsoy and Cardie, 2014) and Recursive Neural Networks (Socher et al., 2011;Socher et al., 2013) to capture global information.