{'fraction_non_alphanumeric': 0.03024085637823372, 'fraction_numerical': 0.005664585191793042, 'mean_word_length': 4.901816267438799, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Classification of Opinions with Non-affective Adverbs and Adjectives

We propose domain-independent language patterns that purposefully omit the aective words for the classification of opinions. The information extracted with those patterns is then used to analyze opinions expressed in the texts. Empirical evidence shows that opinions can be discovered without the use of aective words. We ran experiments on four sets of reviews of consumer goods: books, DVD, electronics, kitchen, and house ware. Our results support the practical use of our approach and its competitiveness in comparison with other data-driven methods. This method can also be applied to analyze texts which do not explicitly disclose aects such as medical and legal documents.


Introduction
Opinion and sentiment analysis has recently received much attention from researchers in the Natural Language Processing (NLP) and Machine Learning (ML) communities. Most of the research addresses the primary clues of sentiments in a binary setting, e.g. positive/negative word orientation (good vs evil), subjective/objective statements (The movie was awesome vs We saw the movie yesterday), texts belonging to positive/negative opinion categories (I recommend this camera . . . vs This book is awful . . . ). In this work, we concentrate on learning opinion from complete texts, classifying the texts as positive or negative.
In opinion learning, NLP and ML research mainly concentrates on the emotional polarity of texts which is expressed through the use of affective words (This is an excellent view shows positive polarity, excellent is an affective word). In this work, we, however, propose that learning opinions should allow for the use of the word categories other than affective. We present a method which uses non-affective adjectives and adverbs (future, full, perhaps), supplemented by degree pronouns and mental and modal verbs, to determine whether a text bears a positive or a negative opinion label.
The method engineers features by using the intrinsic characteristics of a language and avoids extensive and elaborate computational mechanism. Methods used to classify complete texts according to opinions and sentiments usually employ automated feature selection, e.g., [17,15]. Although such methods can be applied to different domains, they sometimes involve complex optimization problems, e.g., N P -hard approximation problems [2].
We concentrate on expressions of stance (maybe, necessary), degree (extremely, any), time (ago, now), frequency (rare, again), size (short, high), quantity (many, few), and extent (big). We show that these indicators reliably represent texts in opinion learning.
We organize the corresponding word categories -stance/degree/time/frequency adverbs, frequency/size/quantity adjectives, degree pronounsinto a semantic hierarchy. Its lowest level works with words; the middle level generalizes word categories into groups; the highest level applies to the text as a whole. The hierarchy avoids the use of emotionallycharged words. We use the hierarchy to extract lexical features from texts. Next, we use the features to represent texts in a series of machine learning experiments.
Empirical evidence obtained on four data sets shows reliability of our approach. The presented method can be applied to analyze texts which do not explicitly disclose affects, e.g., medical and legal documents. This work extends preliminary studies presented in [22]. The rest of the presentation is organized as follows: we introduce word categories used in the test representation, then the hierarchy is presented, followed by description of the information extraction procedure and empirical results; discussion of related work, results and future work conclude the paper.

Text representation
Studies of sentiment and subjectivity analysis mostly concentrate on the use of the affective words in expression of sentiments and opinions. Some opinion studies use topic and domain words and affect-neutral verbs [18,21]. We propose that words which emphasize quantitative properties (high, some), time (old, yesterday) and confidence in happening (can, necessary, probably) can be used in learning opinions.
Such words constitute detailed, specific, description of an object or action [4,9]. We organize them in the following word categories: We use word categories 1 -5 to build the entry level for the hierarchical text representation. To be less domain and topic-dependent, we ignore subcategories closely related to the text topics, e.g., derived (useless), topical (royal,economic), affiliative (American), foreign (ersatz) [4]. We purposefully omit evaluative/emotive adjectives (excellent, disgusting) while constructing the lexical level of hierarchy. This omission allows emphasis on the role of quantitative description in text.

Semantic Hierarchy
In this section, we introduce a hierarchy of text representation. Starting from the bottom, the hierarchy defines the word categories used in detailed descriptions, then groups the categories into four types of comments, and finally combines the types into direct and indirect detailed categories. The levels of the hierarchy are represented by a set of rules which capture the essential characteristics of their language indicators. The rules have the following form: where non-terminal must be replaced by one of the alternatives. Alternatives are composed of other nonterminals and terminals which are the pieces of the final lexical string. The lowest, lexical, level presents terminals for the word categories discussed in Section 2. The middle level organizes word categories into semantic groups. The highest, the most general, level is concerned with text pragmatics.
We determined the list of terminals by finding seed words for these word categories in [4,6] and added their synonyms from an electronic version of Roget's Interactive Thesaurus [20]. To accommodate negative comments, we added the negation rule. There are 303 rule terminals, not counting the negation terminals. Figure 1 shows the rules for finding detailed descriptions in text; within a hierarchy level, the rules are listed in alphabetical order.
We now list some implications of the rules presented by Figure 1: direct Details presents primary clues of quantitative evaluation and attributes of the discussed issues. Two rules of the middle level provide factual information through the word categories of the lowest level: Estimation lists the reference attributes: physical parameters, relative and absolute time.
Quantification expresses the broadness of the discussed reference by specifying its multiplicity, frequency and extent.
indirect Details presents secondary clues of the issue evaluation. Two rules of the middle level define indirect evaluation through the word categories of the lowest level: Comparison presents a comparative evaluation of the discussed issues, their qualities and relations among them.
Confidence reflects on the certainty about the happening of events;

Feature Set Construction
We hypothesize that expressed opinions can be accurately learned from non-affective adverbs and adjectives. To evaluate our hypothesis, we apply the hierarchy to find and extract features for text representation. Our core assumption is the following: the hierarchy rule terminals emphasize important characteristics of the discussed issues. Grammatically, the rule terminals are modifiers, amplifiers and identifiers. In sentences, such words usually precede their references, especially in conversational text [4]. The extraction of words which follow the rule terminals results in the set of words most emphasized in the text. The extraction procedure is presented on Figure 2; it has only one adjustable parameter h, whose value is determined during the empirical step.
To build the set of words most emphasized in the text, we look for words with a high probability of appearance on the right side of the rule terminal. We direct Details Step 1 build a bigram model of the data: 1. for sequences w j−1 w j , j = 1, ..., m, calculate the probabilities of their occurrence in the data; 2. disregard the sequences with the probability of 0; Step 2 find words appearing on the right side of the terminal: 1. for each t i ∈ T , extract bigrams t i w j where the pattern terminals appear on the left side; 2. build the unigram model of the extracted bigrams; 3. remove the terminal unigrams; Step 3 find frequently modified and intensified words:  (1) and (2). The adjustable parameter h is determined empirically. estimate this probability P (w j |T ) by computing: where w j is a word, T is the set of all terminals, t i is a terminal, w j |t i is the event where the word w j appears after t i in text (t i w j ), m is the size of the data vocabulary, n(x) is the number of occurrences of x in the data. The idea behind the search procedure is the following: two-word sequences t i w j -bigrams -which have terminals on their left side capture the modified and intensified words. After extracting such bigrams, we find modified and intensified words. By calculating the probability of the word occurrence after a terminal, we can find most frequently modified and intensified words. Concentrating on one-side bigrams prevents the multiple extraction of the same word.
In supervised learning experiments, each text is represented by a vector x 1 , . . . , x n , y, where x i is a number of occurrences a word w i , a feature, appearing in the text, and y is the opinion label. As a weighting factor, we use normalization of the vector attributes with respect to the number of words in the text. It eliminates the bias introduced by the length of the text. Based on the rule terminals and the extracted words, we construct three feature sets for text representation: I terminals of direct Details rules enhanced by personal pronouns; h was determined by frequencies of personal pronouns; II terminals of all the hierarchy rules enhanced by the most frequent extracted extracted words; h was determined by frequencies of personal pronouns; III the terminals and all the words extracted by the procedure presented in Figure 2; the cut-off threshold h = 5 was determined by using Katz smoothing to ensure reliability of data representation.

Empirical results
We ran experiments on data introduced in [ For this study, we extracted the review texts; see Figure 3 for samples of the extracted reviews; in those texts we have marked the features presented by the hierarchy ( Figure 1) and constructed through the procedure (Section 4) . Correspondence among information provided by different fields is left for future work. Review texts are long enough to provide meaningful communication and lexical information; Table 1 lists the descriptive statistics of the extracted data. These four sets allow us to compare our empirical results with those obtained by other methods. In order to establish how a speaker's detailed descriptions are related to her opinion, we apply supervised learning techniques that construct a function on a set of input and output pairs ( x, y) where x represents a text and y is its opinion label (training data). This function is then used to predict opinion labels on previously unseen examples (testing data).
We want to establish a link between information extracted by patterns and the text opinion categories, e.g. positive or negative. We expect non-linear dependencies between the terminals' appearance in texts and a speaker's opinion. We applied decisionbased C4.5, prototype-based k-Nearest Neighbor and kernel-based Support Vector Machine (SVM). SVM performed considerably better than other algorithms on all the four data sets. The algorithm is known for its high accuracy in text classification. SVM does not make any assumption about the data distribution and could work with non-linear dependencies, albeit on one level of learning. Further we report only the SVM's results. We use the Weka's implementation 1 . Classification measures use the following counts: Data class Classified as pos Classified as neg pos tp fn neg fp tn Accuracy = tp + tn tp + fn + fp + tn evaluates the overall performance of svm; tp and tn provide a detailed analysis of the algorithm's performance on positive and negative classes. We use tenfold cross-validation to compute the three measures because of its generalization accuracy and the reliability of its results.
We compare text representations built on the three levels of rules presented by Figure 1. Table 2 reports learning results obtained on the three representations introduced in Section 4. As the baseline, we apply SVM on texts represented by the feature set I . All 62 selected words appear frequently in the data and provide substantial information about texts. These features include adverbs of degree and adverbs of frequency which were used by [3] in sentiment classification. Adding all II features makes a statistically significant difference in accuracy (paired t-test,

Book Reviews
Extracted review sample Label This thing sat on my shelf, half-read for the longest time. Only the notice of the upcoming release this November of Pynchon's next got me motivated enough to dig into it again. It's not that it isn't brilliant. No one else around can dazzle you with so much wit and wonder. The first encounter with the talking dog is as magical as anything you'll ever read. And it's not like this is the only Pynchon novel that takes some effort to get into. pos I picked up the first book in this series (The Eyre Affair) based purely on its premise and was left somewhat underwhelmed. Still, the potential for the series seemed so large that I went ahead and read this second one too . . . He just looks away from where the spray emits-and barks again! It also doesn't work 100 % of the time...and we're not sure why. When we fill it, it seems to work fairly well right after but it either does not have as many sprays as it is supposed to, or it isn't working very long. It does work well for my other small dog who is not such a persistent barker. Terriers are just too stubborn to care if they're getting sprayed, I guess.

Kitchen and Houseware Reviews
neg Fig. 3: Samples of the extracted reviews. Terminals of rules in Figure 1 are in bold; the words found by procedure in Figure 2 are in italics; negations are ignored if they do not appear before the rule terminals. For each data set, the upper sample has a positive opinion label that labels the whole text whereas the lower sample has a negative opinion label.
When we use all the extracted words, the statistically significant difference increases (paired t-test, P = 0.003). For each representation, the classification results are close across the data sets. However, there is a remarkable difference between the Electronics data and the three other sets. Let's consider true classification of the positive and negative reviews. For Books, DVD, Kitchen sets, the positive reviews are always classified more correctly than the negative reviews (the only exception is a tie for Books on the II representation ). The Electronics set provides the opposite results: the negative reviews are always classified more correctly then the positive ones. It is interesting to observe that only 303 rule terminals, i.e., the II features, already provide an opinion accuracy of 74% − 78%. These are reliable results for opinion learning, since human agreement on whether a message provides a positive or a negative opinion about the discussed topic could be 78% for positive opinions (tp) and 74% for negative opinions (tn) [16]. On the four data sets, all extracted features provide accuracy of more than 80%. There are 1999 terminals and extracted words used. Reported tp and tn show that the extracted features provide a wellbalanced classification of the classes: on all the four data sets difference between tp and tn is < 10%. This can be only achieved if the algorithm is successful in learning both positive and negative classes.

Related work
Opinion and sentiment analysis that focuses on whether a text is subjective, bears positive or negative opinion or expresses the strength of an opinion has received a vast amount of attention in the recent years. In this section, we discuss only research with application to complete texts, which sometimes are referred to   as documents. We omit research on sentiment/opinion analysis of terms, phrases, sentences and other text segments; references and discussion can be found in [7]. Some of this work relied on a list of characteristics of reviewed products. Hu and Liu extracted features based on association rule mining algorithms in conjunction with frequency to extract main product characteristics [10]. These characteristics are then used to extract adjacent adjectives which are assumed to be opinion adjectives. Later, these opinion adjectives are used to find product characteristics that are mentioned only once or few times. In contrast, we opted for a domain-independent method that does not involve the use of the domain's content words. Popescu and Etzioni (2005) extracted product characteristics from noun phases in the data and matched them with known product features. In contrast, we opted for a domain-independent method that does not involve the use of the domain's content words.
For automating recognition and the evaluation of the expressed opinion, complete texts are represented through N -grams or patterns and then classified as opinion/non-opinion, positive/negative, etc. [19]. In [5], the authors combine supervised and semisupervised structural correspondence learning to classify the four data sets. They use fully automated feature selection based on frequency and the mutual information of words. However, the difference in evaluation technique does not allow us to directly compare the obtained results.
Syntactic and semantic features that express the intensity of terms are used to classify the text opinion intensity [23]. Benamara et al. studies the impact of combining adverbs of degree with adjectives for the purpose of opinion evaluation [3]. Our approach deals instead with opinion analysis which is broader than the analysis of sentiments. We focus on the formalization and utilization of non-emotional lexical features.
Except sentiment analysis, machine learning is used to study opinions from the point of view of predictive power [12], strength [23], and also in summarization and feature extraction studies [8]. Although Kim and Hovy (2007) generalized bi-and trigrams found in texts (e.g. NDP will win and Liberals will win became Party will win), they did it bottom-up, without providing theoretical background. We, instead, used a top-down hierarchical approach based on pragmatic and lexical rules. Wilson et al (2006) concentrated on learning subjective vs objective sentences and sentence clauses. Their initial manual clues included verbs of judgment (reprove, vilify); in the final text representation they use syntax clues. In contrast, to represent texts, we look for non-affective adverbs and adjectives.
Consumer and expert-written product reviews are intensively studied by psychology, marketing, etc. [14]. Kamakura et al. analyzed movie reviews written by 46 experts [11]. They were one of the first to do research on the information contained in unguided reviews and built a model linking the expert's history of movie evaluations and quantitative information found in reviews. However, the authors did not actually analyze the texts or language cues contained in the reviews. Their results showed that experts are not uniformly informative across movies that they scored differently. This study supports our views that specific information in product reviews relates to the speaker's opinion, although their results were obtained on expertwritten reviews.
The research listed above does not consider a hierarchy of opinion disclosure. A pragmatic-lexical hierarchy of semantic verb categories was proposed in our previous work [21]. We showed that the hierarchy worked well in environment where negative opinions were expressed indirectly, without the use of negative adjectives or adverbs, e.g. debates in the US federal Senate. In contrast, in the current work, we concentrated on the use of non-affective adverbs and adjectives and degree pronouns.

Conclusion and future work
We have proposed a hierarchical text representation (the highest level that considers a text as a whole) and derived rules (the middle level), as well as the rules' terminal categories (the lowest level that works with words). The terminals were used to extract the emphasized information from the text. Our goal was to build domain-independent rules that do not rely on domain content words and emotional words. Further, the additionally extracted words and the rule terminals were used to represent texts in learning experiments.
In our experiments, we used four data sets which texts were gathered in different domains. We studied the relevance of detailed, specific words to the learning of positive and negative opinions. Our empirical results show that the corresponding lexical features are effective in learning opinions.
Our approach can be applied to analyze language in texts which traditionally lack emotive and affective words. Medical and legal domains provide us with such texts. It is worth note that these two domains attract more and more attention of Text Data Mining community as evidenced by many publications, for example, in the Journal of the American Medical Informatics Association 2 and the International Journal of Law and Information Technology 3 . Another venue can be a joint analysis of information contained in different fields of reviews.
In this study, we applied supervised learning algorithms that required labeled data, which is usually restricted. Our future work will be to incorporate unlabeled data and apply semi-supervised approaches, e.g., a framework of learning predictive structures [1]. A notable drawback of this framework is the need to find "good" features for data representation. We can overcome this problem by using the results of the current study. Another direction worth trying is to incorporate more pragmatic knowledge into feature construction, e.g., likelihood of the use of features in written or spoken language obtained from the British National Corpus [13].