{'fraction_non_alphanumeric': 0.02743677878713479, 'fraction_numerical': 0.011232506751780015, 'mean_word_length': 5.21395881006865, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Attention-based Bidirectional LSTM for Deceptive Opinion Spam Classification

Online Reviews play a vital role in e commerce for decision making. Much of the population makes the decision of which places, restaurant to visit, what to buy and from where to buy based on the reviews posted on the respective platforms. A fraudulent review or opinion spam is categorized as an untruthful or deceptive review. Positive reviews of a product or a restaurant helps attract customers and thereby lead to an increase in sales whereas negative reviews may hamper the progress of a restaurant or sales of a product and thereby lead to defamed reputation and loss. Fraudulent reviews are deliberately posted on various online review platforms to trick customers to buy, visit or distract against a product or a restaurant. They are also written to commend or discredit the product's repute. The work aims at detecting and classifying the reviews as deceptive or truthful. It involves use of various deep learning techniques for classifying the reviews and an overview of proposed approach involving Attention based Bidirectional LSTM to tackle issues related to semantic information in reviews and a comparative study over baseline machine learning techniques for review classification.


I. INTRODUCTION
With the advent of the internet, a large depot for digital information and user-generated content is available. Such content is generated through community interactions, papers, documents, opinions expressed on various platforms, social media websites and many other sources. Content like reviews, helps businesses improve their sales and grow. With an increase in reviews on various review platforms, customers rely on more on online reviews for decision making. Also, clients use online reviews for decision making in context to their future business partners. Thus, online reviews may help businesses grow or harm their reputations. Opinion spams are such online reviews that either harm or help businesses grow by posting fake reviews on the online review platforms. Detecting opinion spams or fraudulent reviews manually is slog altogether. Human beings are not always able to classify truthful and fake reviews. Current research has questioned the credibility of online reviews. For instance, around 20% of reviews posted on Yelp are found to be faked by paid human writers [1]. Opinion spamming has opened easy monetization. Spam reviews can be classified into two types: [4] Misleading reviews and non reviews. Misleading reviews are either undeserving positive reviews or unjust negative reviews regarding a product or a service. Non reviews are those which contain no opinions about the product, rather, they either promote irrelevant advertisements or comments.
There has been ample research in developing review spam classification systems. Past researches in text classification make use of labeled data for text classification. Awad and Elseuofi applied various machine learning techniques for the automated classification of emails [2]. Manlangit et al. proposed an intelligent system which could detect fraud transactions using classification algorithms [3]. One of the major issues faced for developing such classifier systems for review platforms is the imbalance in data. The frequency of spam samples can be sparse in comparison to non-spam samples. Conventional Machine learning algorithms when trained on such data can have a bias towards majority class which will lead to a new sample being accepted as relevant. This ambiguity can be resolved using balanced dataset. This can be achieved using either undersampling the majority class or oversampling the minority class.
In existing research, comparison of results is limited due to consideration of different features, models or data. In this paper, an approach is proposed that involves deceptive reviews which are classified using a supervised learning approach on a balanced data set. The classification is treated as a text classification problem. The reviews are classified into two classes -Deceptive and Truthful. The work makes use of labeled data to plot a decision boundary based on which a new test case can be classified into the appropriate class. Text preprocessing techniques and different models are employed to successfully detect review spams. This includes deep-learning models that recently produced state-of-the-art benchmarks on text classification problems. Along with this, word embeddings are used to train continuous word representations. These pretrained word vectors improve the performance of the deep learning models, but results show a linear classifier based on frequency-based embeddings outperforms the other approaches along with the deep learning models.
The remainder of the paper is structured as follows: Section 2 provides an overview of related work. Along with the overview of Cleaning and pre-processing techniques used prior to classification and the proposed approach is described in Section 3. Description of the data set, evaluation metrics and experimental results are provided in Section 4. Finally, Section 5 draws the conclusion and suggests future direction of the work.

II. RELATED WORK
Fake Review Detection has involved various approaches based on review content, user-reviewer behavior, and sentiment analysis in classifying user reviews. Review classification problem was first put forth by Jindal and Liu [4]. They classified reviews into three categories: [4] Type 1: False Opinions: undeserving positive reviews or unjust negative reviews. Type 2: Reviews on Brand: Reviews on the brand of the product instead of the product itself. Type 3: Non-Reviews: They are texts with random content like an advertisement which is irrelevant to the product. Jindal and Liu have suggested that outlier reviews must be identified as suspicious reviews [4]. Yoo and Gretzel (2009) gathered 40 truthful and 42 deceptive hotel reviews and manually compared the linguistic differences between them. [5]. Ott et. al (2011) employed Turkers to write fake reviews and created a benchmark data set [8]. Their data set was adopted by the further line of work [9] Feng et al. (2012) looked into syntactic features from context-free grammar parse trees [11] to improve the classification performance. Recently, deep neural networks have shown promising results in text classification problems. They have shown good results in learning underlying features in literature. Most characteristic progress has been made by Zeng et Al.(2014), who used Convolutional Neural Networks for relation classification [16]. Since CNN doesn't consider the context of the sequence and also it considers words only in close range, another approach is suggested by Mikolov et Al.(2010) based on Recurrent Neural Networks [17]. Another related work proposed by Zhang and Wang (2015) involved use of Bidirectional RNN [18]. But Bidirectional RNN faces a vanishing gradient problem. Hence, to resolve this problem, Long short term Memory was introduced by Hochreiter and Schmidhuber (1997) [21].
Finally, proposed work is based on and related to three approaches -one architecture involving Bidirectional Long short term memory proposed by Zhang

III. PROPOSED APPROACH
Classification of reviews can be done via two categories: Review centric features and Reviewer centric features. The proposed approach focuses on review centric features. Classification of reviews is done using neural network model which learns continuous vector representations for documents of variable lengths, which is used as features to classify a spam or not for each document. Also, comparative study of methods of text classification for opinion spam.

A. Preprocessing
Data obtained from online review platforms cannot be fed directly into the machine learning algorithm. To extract useful information from the data, different preprocessing techniques need to be applied on the data set. The textual data in Deceptive Opinion Spam Corpus [9] The above mentioned preprocessing techniques are implemented using the nltk, beautiful soup and scikit learn packages in Python 3.6. After carrying out these techniques, the text data is converted to a sparse matrix consisting of numbers which can be fed to a classification algorithm. As a part of the evaluation process,the data is split into train and test sets with the model being trained on 80% of the data and results are calculated for the remaining 20% of data set. For any new test case, the above preprocessing methods still need to be carried out and the class label is predicted for the feature vector of the test case.

B. Word Representations
Word representations are either fixed or distributed. In this paper, distributed word representations are used. Distributed word representations represent word as features vectors. They show semantical or syntactical dependencies. For instance, 'car' and 'automobile' can have similar vector values and low numerical difference since they have related meanings. Hence, useful properties can be obtained using such linear relationship. Word representations used in the work: • Word2Vec: Word2Vec model is composed of shallow neural network models called CBOW (Continuous Bag of Words) and Skip-Gram; used to learn word embeddings [24]. This model constructs vocabulary from training data set and then prepares word embedding vectors. Pretrained vectors can be loaded published by Google trained on part of Google News Data set. The model consists off 300-dimensional vectors for 3-million words and phrases. • GloVe: GloVe is an unsupervised algorithm for obtaining word embedding vectors [12]. Training is performed on grouped global word-word co-occurrence statistics from a corpus, and the resulting representations showcase linear substructures of the word vector space. [12] GloVe has been used to generate word vector.

C. Model Building
The next step is to train the classifier using the features created in previous step. Different machine learning models can be used to train the final model. Different classifiers are implemented for this purpose and compare their results with performance metrics and choose the final model for the classification. Following are the classifiers implemented: -Bidirectional LSTM -Recurrent CNN -Bidirectional LSTM Attention 1) Naive Bayes Classifier: Naive Bayes illustrates a supervised learning technique. It is a classification technique based on Bayes' Theorem. It assumes independence among predictors. In layman's terms it assumes presence of particular feature in class is unrelated to the other features' presence. Naive Bayes model is easy to build and performs well on large data sets. It is known to outperform sophisticated classification models.
2) Linear Classifier -Logistic Regression: Logistic regression is a linear classifier which measures the relationship between the categorical dependent variable and one or more independent variables. It estimates the probabilities using a logistic function.
3) Support Vector Machine: Support Vector Machine exemplifies supervised learning technique used for both classification and regression which finds the hyperplane that best segregates two classes.
4) Stochastic Gradient Descent: Stochastic Gradient Descent is a efficient approach to discriminative learning of linear classifiers like SVM and Logistic Regression under convex loss functions. SGD has been successfully used on large data sets and it's easy implementation and efficiency has helped solve sparse machine learning problems in text classification and natural language processing. Naive Bayes Classifier, Logistic Regression, Support Vector Machine, Stochastic Gradient Descent are trained on features such as count vectors, word level TF-IDF vectors, N-gram level TF-IDF vectors and Character level TF-IDF vectors and calculate the performance metrics for further comparative study and abalative analysis. 5) Deep Neural Networks: Deep Learning models have been successful in natural language processing tasks like distributed word learning, sentence and document representation [23], parsing [22], statistical machine translation [25], sentiment classification [26] [7], opinion-spam classification [9] [10] [8].
In this paper, the focus is on opinion spam classification in online reviews using deep learning based approaches. Proposed method is compared with widely used text classification methods and the state-of-the-art approaches is done. The approach presents six different architectures to achieve this. Then exploratory analysis is performed and a comparative study over previous architectures and architectures like Attention-CNN, Attention Bidirectional LSTM and a hybrid of both approaches as Bidirectional LSTM-CNN-Attention architecture is done. These architectures have shown promising results in the other text-classification tasks [7], [26], [27], [30]- [32]. Thus, we try to implement these architectures alongside the proposed architecture.
• Convolutional Neural Network: Text classification using embeddings and Convolutional Neural Networks was previously introduced by Yoon Kim [7]. Similary, CNN is used along with GloVe 100-Dimension embeddings. The convolutional kernel concatenates the embeddings into predefined window to compute the output. Subsequent layers apply filters and combine the output for the classification.

IV. EVALUATION A. Dataset Description
To study and compare results of various approaches benchmark dataset Deceptive Opinion Spam Corpus consisting of 1600 hotel reviews created by Ott et. al [9] [10] [8] was used. This corpus contains: • 400 truthful positive reviews from TripAdvisor (described in [8]) • 400 deceptive positive reviews from Mechanical Turk (described in [8]) • 400 truthful negative reviews from Expedia, Hotels.com, Orbitz, Priceline, TripAdvisor and Yelp (described in [10]) • 400 deceptive negative reviews from Mechanical Turk (described in [10]) Each of the above datasets consist of 20 reviews for each of the 20 most popular Chicago hotels [8]. The dataset consists of features as hotel, polarity, source, text and target value as deceptive.

B. Performance Metrics
The classification by linear models is evaluated by the measure of accuracy, precision, recall and F1 Score.

Accuracy =
T P + T N T P + T N + F P + F N Precision = T P T P + F P

Recall =
T P T P + F N

D. Evaluation of Results
The training dataset is passed through various classification algorithms namely: Multinomial Naive Bayes, Logistic regression, Stochastic Gradient Descent and Support Vector Machine. The results based on metrics like accuracy, precision score, recall and F1 score are presented in Table 1. SVM: After being trained on textual data, it gives the least F1 score of 68.35%, least accuracy of 56.25%, least precision of 52.5% and highest recall 97.92% SGD and MultinomialNB: The accuracy when considering text feature is observed to be 90% for Multinomial Naïve Bayes and around 88% for Stochastic Gradient Descent. From table 1, it can be clearly concluded that the Multinomial Naïve Bayes classifier performs better than the other classification algorithms. Logistic Regression too performs better with a accuracy of 87% and F1 score of 86.45% Amongst all deep learning architectures used Recurrent Convolutional Neural Network architecture and Attentionbased Bidirectional Long Short Term Memory show better results with validation accuracy of 92.19% and 90.25% respectively. It has better results than most of the existing approaches involving Multinomial Naive Bayes, Logistic regression, Stochastic Gradient Descent and Support Vector Machine and other deep learning architectures.

V. CONCLUSION
It was observed that classification algorithms achieve satisfactory results on review-centric features like n-gram TF-IDF vectors and character level TF-IDF vectors through the abalative analysis. Experimental results demonstrate that the model performs significantly better than previous methods using deep neural networks. Visualization of the attentionbased approach adapted from [13] demonstrates that the model is effective in capturing important words and sentences. The performance of the approach can be enhanced for all deep learning architectures through hyper-parameter tuning.