{'fraction_non_alphanumeric': 0.026066706730769232, 'fraction_numerical': 0.011268028846153846, 'mean_word_length': 4.953935599284437, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
An Unsupervised Approach for Aspect Category Detection Using Soft Cosine Similarity Measure

Aspect category detection is one of the important and challenging subtasks of aspect-based sentiment analysis. Given a set of pre-defined categories, this task aims to detect categories which are indicated implicitly or explicitly in a given review sentence. Supervised machine learning approaches perform well to accomplish this subtask. Note that, the performance of these methods depends on the availability of labeled train data, which is often difficult and costly to obtain. Besides, most of these supervised methods require feature engineering to perform well. In this paper, we propose an unsupervised method to address aspect category detection task without the need for any feature engineering. Our method utilizes clusters of unlabeled reviews and soft cosine similarity measure to accomplish aspect category detection task. Experimental results on SemEval-2014 restaurant dataset shows that proposed unsupervised approach outperforms several baselines by a substantial margin.


Introduction
User-generated reviews are valuable resources for both companies and potential customers. These reviews can help online retail companies in recognizing their weaknesses and strengths, and facilitate the decision process for consumers. However, it is impractical and tedious to read such a huge amount of reviews one by one manually. Therefore, the need for an automatic system that processes a huge amount of reviews and provides a summary of these different reviews in a suitable form seems necessary.
Given a list of pre-defined aspect categories (e.g. 'food' and 'price' in restaurant domain), aspect category detection aims to assign a subset of these categories to review sentences. SemEval 2014 Task 4 [9] tried to tackle this task by providing datasets for multiple domains, including restaurant and laptop reviews.
Previous works in the literature mostly are based on supervised machine learning approaches [3][6] [2]. Although these methods perform well, they need annotated train data which are time-consuming and expensive to obtain. Thus, unsupervised approaches maybe be a good choice, especially for low resource languages.
Soft cosine measure is a similarity measure that assesses the similarity between two sentences, even when they have no words in common [15]. For aspect category detection, our method utilizes soft cosine similarity. Firstly, we cluster a set of unlabeled review sentences into k cluster. Clustering is performed based on the Euclidean distance between the average of their word embeddings. Our motivation for using the cluster of sentences is based on the intuition that sentences in the same cluster share similar information about categories they belong to. The similarity between a given sentence and a pre-defined category is defined as the soft cosine similarity between sentence and a set of manually selected seed words corresponding to that category. So, the similarity values can give us information about categories that sentence belongs to. We also define similarity between a cluster and a category as averaging the similarity scores of the sentences in the cluster. Finally, given a test review sentence, scores obtained for the sentence and the nearest cluster to it are interpolated. These final scores are normalized and used to detect the categories mentioned in the sentence. If the similarity of a category surpasses a threshold, it is assigned to the sentence.
We evaluate our method on SemEval-2014 restaurant dataset. Experimental results show that the proposed method outperforms several baselines by a high margin.

Related works
Early works for addressing aspect extraction relied on approaches such as identifying frequent nouns and noun phrases using association rule mining, dependency relations, and lexical patterns [5][11] [10]. The SemEval workshops, over the course of three years, has included aspect-based sentiment analysis in their competitions. One of the subtasks introduced during SemEval is aspect category detection, which our proposed method is going to address. Most of the supervised approaches proposed to address this subtask utilizing machine learning algorithms and train a set of the one-vs-all classifier using hand-crafted features [6], [17], [16].
There are only a few unsupervised approaches to address the aspect category detection subtask in the literature. In [4] authors trained a network similar to auto-encoder with attention mechanism to attend to aspect-related words. [8] propose to use a double propagation technique to mine rules based on dependency relations for finding aspect terms of each category. Also, irrelevant aspect terms were pruned using stop words and the PageRank algorithm on a graph-based approach. [14] proposed an unsupervised method called 'spreading activation' that performs association rule mining, using a set of seed words and a co-occurrence matrix between words to form a co-occurrence digraph to detect aspect categories. One of the shortcomings of this method is the need for tuning multiple parameters. Unlike this method, our proposed method only re-quires a small number of parameter (a threshold, the number of clusters, and the interpolation coefficient).

Proposed Method
This section describes our proposed unsupervised method. In the following subsections, we will discuss each of the main components in detail.

Manual Selection of Category Seed Words
We manually select a set of 5 seed words for each category (20 in total) to represent the category. Because the anecdotes/miscellaneous category is very unspecific and abstract, we didn't choose any seed words for it [14]. For this aspect, following [14], we would assign this category only to sentences that were not assigned any other category. The set of seed words for each category can be seen in Table 1.

Sentence similarity
In order to find the similarity score of a given sentence compares to a category, we utilize soft cosine measure. For each category, we define the similarity of the given sentence to that category as the average of soft cosine similarity values between the sentence and each of the seed words belonging to that category. Let x be a given sentence and ai be the i-th category. We define the sentSima i (x) to be the similarity value between ai and x as seen in equation 1.
As stated in [18], the sigmoid function can have a discriminating effect on the similarity values obtained from similarity measures like cosine similarity. To make the similarity values more discriminating, we transferred the similarity values obtained in the previous step via the sigmoid function as seen in equation 2.
Now, for each sentence we have a vector sentScore ∈ R c , which c is the number of categories and each element represents the similarity score between the sentence and a pre-defined category.

Cluster similarity
A set of unlabeled sentences are acquired from the Yelp dataset challenge 1 . In order to decrease noise samples and since the precision is a more important factor than recall in acquiring true sentences, only sentences that contain at least one of the category names (eg. 'food', 'service') are selected. Using k-means clustering algorithm, these unlabeled sentences are clustered into k clusters. Clustering is done based on the Euclidean distance between the average of word embedding of sentence words, where word embeddings are trained on the Yelp dataset using Continuous Bag of Words (CBOW) algorithm. For each cluster, a similarity value per category is calculated as shown in equation 3: where c k is the k-th cluster. Similar to sentence similarity, we also utilize sigmoid function in here for it's discriminating effect.
Therefore, for each cluster we have a vector clustScore ∈ R c , which c is the number of categories and each element represents the similarity between cluster samples and one of the pre-defined categories.

Assign aspect categories
Given a test review sentence, first, we calculate the sentScore vector for the sentence, then, this vector is interpolated with the clustScore corresponding to the nearest cluster to the given sentence, as shown in equation 5. The nearest cluster is found by finding the closest centroid to the sentence based on the Euclidean distance between the average of the word embeddings of the sentence and the cluster centroid.
where sentScoreN and clustScoreN are L2-Normalized vectors of sentence and its cluster respectively. Categories that exceed a threshold are assigned to the sentence. We find the optimal threshold by a simple linear search.

Dataset
We conducted our experiments on the SemEval 2014 Task 4 benchmark dataset in the restaurant domain. The aspect categories are as follows: food, price, service, ambience, anecdotes/miscellaneous. There is a total of 800 sentences in the test data, and each one contains at least one of the categories. In order to gather a set of unlabeled indomain sentences, we randomly pick 10000 sentences that contain at least one of the pre-defined category names from the Yelp dataset challenge.

Experiment Setup
We train word embeddings on the Yelp challenge unlabeled dataset. Word embedding size is set to 300 and CBOW model [7] with default parameters is used for training the embedding vectors. Removing stop-words and tokenizing sentences are performed as the pre-processing step using NLTK package [1]. For implementing soft cosine similarity measure and word embedding training, we used gensim package [12]. A simple linear search is performed to find the optimal hyperparameters α and K -the interpolation coefficient between sentence score and its cluster score, and the number of clusters for k-means clustering respectively. We set the parameter α to 0.7 and the number of clusters to 17 to obtain the best result in our method.

Results and Analysis
Micro F1-score, precision, and recall of all the category labels are used as evaluation metrics and performance of the proposed method evaluated on test data from SemEval-2014 [9]. The test data contains 800 test review sentence in the restaurant domain. The proposed method is compared to the following baselines. Random baseline assigns a random category based on the frequency of the categories appearing in the train data. Majority baseline assigns the two most common categories in the train data ('food' and 'anecdotes/miscellaneous') to each of the test sentences. COMMIT-P1WP3 [13] baseline is a co-occurrence based method. After constructing a co-occurrence matrix between the words and the pre-defined categories, this method calculates the probability of a given sentence belonging to each category based on this matrix. V3 [8] baseline utilizes WordNet similarity to compare the detected aspect terms to representative terms of each category. SemEval-2014 baseline [9] is the baseline provided by the SemEval 2014 which uses a simple k-nearest neighbor classifier to detect aspect categories. In the Spreading Activation baseline [14], using a set of seed words and a co-occurrence matrix similar to [13], a set of association rules are mined, associating categories to terms appearing in the data. The aspect category detection task is then performed using these association rules. Table 2 shows that among baseline methods, Spread Activation, an unsupervised approach, performs the best with F1-score 67%. Our unsupervised method outperforms Spread Activation by 9.98% in terms of F1-score and similarly improves over SemEval-2014 baseline, V3, COMMIT-P1WP3, Majority, and Random baselines by 13.08%, 16.78%, 17.68%, 27.31% and 47.30% respectively. These results clearly demonstrate the effectiveness of the proposed unsupervised method in aspect category detection task. Figure 1 (a) plots the sensitivity of our system to the interpolation coefficient α in equation 5. According to the curve, to achieve the best performance a higher weight should be given to sentence scores itself, which indicates that it plays the main role in finding corresponding categories. The results of the experiments prove our intuition that interpolating the cluster scores and the sentence scores improves the classification performance. The reason behind this improvement can be the sentences that contain a seed word or a semantically close word to the seed words of a category. Such sentences will improve the scores of other sentences in the cluster for that category. The best   result was obtained at α = 0.7. We also conducted experiments on the number of clusters in the k-means clustering algorithm. Figure 1 (b) provides the results of these experiments. We found the optimum value for k to be 17. As can be seen in the figure, the result of our method tends to deteriorate with k larger than 17. This can be due to the lowering consistency of clusters with larger k. Also, for the k smaller than 9, the same thing can happen.

Conclusions
In this paper, we propose an unsupervised approach for aspect category detection that utilizes soft cosine similarity and the well-known k-means clustering to detect categories belonging to a review sentence. Experimental results on SemEval-2014 benchmark dataset show the effectiveness of our method compared to several supervised and unsupervised baseline methods.