{'fraction_non_alphanumeric': 0.034229973396393734, 'fraction_numerical': 0.016080402010050253, 'mean_word_length': 4.908487600419141, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Yelp Review Rating Prediction: Machine Learning and Deep Learning Models

We predict restaurant ratings from Yelp reviews based on Yelp Open Dataset. Data distribution is presented, and one balanced training dataset is built. Two vectorizers are experimented for feature engineering. Four machine learning models including Naive Bayes, Logistic Regression, Random Forest, and Linear Support Vector Machine are implemented. Four transformer-based models containing BERT, DistilBERT, RoBERTa, and XLNet are also applied. Accuracy, weighted F1 score, and confusion matrix are used for model evaluation. XLNet achieves 70% accuracy for 5-star classification compared with Logistic Regression with 64% accuracy.


Introduction
Online reviewing for businesses becomes more and more important nowadays, where customers can publish their reviews for businesses, and other potential customers or shop owners can view them. Positive feedback from customers may prosper the store businesses, while negative one could have opposite consequences. Yelp, one of the largest company founded in 2004 for publishing crowd-sourced reviews about businesses, provides one open dataset, Yelp Open Dataset [1], which has tremendously many data about businesses, reviews, and users. Such dataset has been proven to be a good material for personal, educational, and academic purposes.
Among multiple tasks on the Yelp Open Dataset, predicting ratings for restaurants based their reviews is one of fundamental and important tasks. This task can help Yelp classify reviews into proper groups for its recommendation system, detect anomaly reviews to protect businesses from malicious competitions, and assign rating to texts automatically.
Yelp review rating prediction can be done in multiple ways, such as sentiment analysis and 5-star rating classi-fication. In this paper, we will focus on rating prediction for restaurants based only on their review texts. This task can be viewed as a multiclass classification problem, where the input is the textual data (reviews), and output is the predicted class (1-5 stars). We will apply both machine learning and deep learning models. After analyzing data distribution, splitting datasets, and extracting features, we will use four machine learning methods, including Naive Bayes, Logistic Regression, Random Forest, and Linear Support Vector Machine (SVM) [8]. Then we will focus on four transformer-based models, including BERT [4], Dis-tilBERT [16], RoBERTa [10], and XLNet [20], where several different architectures will be tried with hyperparameter tuning. This project is done on Google Colab, where multi-processors and GPUs are available. The code is publicly available at GitHub 1 .

Related Work
In past years, many projects have been done on the Yelp Open Dataset. The data distribution has been analyzed thoroughly in [7,3]. The review rating prediction has also been performed. Multiple feature generation methods and several machine learning models including Naive Bayes, Logistic Regression, Support Vector Machine (SVM), and Gaussian Discriminant Analysis have been used for this classification task in [6,5,2,21]. One reported best accuracy for the 5-star classification is 64% on the testing set in [5]. Some deep learning model such as Neural Network, Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), and Bidirectional Encoder Representations (BERT) were also applied in [13,9].

Dataset and Preprocessing
We will use Yelp Open Dataset [1] in this paper. This Yelp dataset is a subset of Yelp business data, which supplies a good material for academic and research projects.
Ideas and methods from machine learning, deep learning, and natural language processing (NLP) can be built and tested on this dataset. This dataset includes 8,021,122 reviews from 209,393 businesses in 10 metropolitan areas. These data are structured in JSON files, including business, review, user, check-in, tip, and photo data. We will only use business and review data.

Data Preparation
In order to obtain more accurate rating prediction, we will only use businesses data in restaurant category. We obtain 63,944 restaurants business IDs from business.json. Then, their 5,055,992 reviews are extracted from review.json. Some samples of data are shown in the Table 1. Rating stars are between 1 star (worst) and 5 stars (best).

Data Distribution
The numbers of reviews from each rating category are not equal. Such data distribution is shown in the Table 2 and Figure 1. We can find the data are skewed towards two polarities, and majority (65.9%) reviews are in 4 and 5 stars. This imbalanced distribution was verified in [3]. Besides, the numbers of tokens in each reviews are also imbalanced. The histogram of the number of tokens is shown in the Figure 2. The distribution shows that 73.1% reviews are no longer than 128 tokens, and 92.8% reviews are no longer than 256 tokens.

Dataset Splitting
In order to deal with the imbalanced raw data, we need to reconstruct a balanced training dataset. Thus, 1,250,000 reviews are resampled from the raw data for a training dataset, where 250,000 reviews are from each category (1-5 stars). Besides, both validation and testing datasets have 250,000 reviews, but they roughly follow the imbalanced distribution of the raw data.

Feature Extraction
With the purpose of using the machine learning models, we firstly build features by converting textual documents to numerical data (matrices). Some detailed text preprocessing pipelines were given in [9,2]. We will use vectorizers from scikit-learn [12] with the following settings: • Implementing binary and integer version Count Vectorizer and Tf-Idf Vectorizer for word representation, • Using unigram and bigram, • Setting minimum document frequency as 5, • Converting all characters to lowercase, • And removing English stop words.
Count Vectorizer can convert text documents to a token counts matrix, while Tf-Idf Vectorizer will used tf-idf in-stead of token counts. Tf-idf is a product of term-frequency tf(t, d) and inverse document-frequency idf(t): where tf(t, d) is the frequency of term t in the document d, and idf(t) is defined by where n is the total number of documents in this document set, and df(t) is the number of documents in the document set that contain term t. For the binary version vectorizer, all non-zero counts are set to 1. Sparse matrices will be returned from these vectorizers. In order to compare the performances of these vectorizers, we test their generated feature matrices with logistic regression. The comparison results on the validation set are shown in the Table 3. The results show that the integer version Tf-Idf vectorizer has the best performance on the validation set, which will be used for machine learning models. For the transformer-based models (e.g. BERT), we use textual data directly without handmade features. More further descriptions will be presented in the transformer-based models section.

Experiments and Results
In this section, we will apply machine learning and deep learning models for rating prediction on Yelp review. Evaluation metrics will be introduced first, then four machine learning models and four transformer-based models will be implemented for this task. Comparisons and conclusions will be derived at the end.

Evaluation Metrics
In order to evaluate machine learning and deep learning models in a fair way, we will use the following evaluation metrics: • Accuracy classification score: the fraction of correctly classified samples.
• Weighted averaged F 1 score [15]: where L is the set of labels, y l is the subset of true y with label l,ŷ l is the subset of predictedŷ with label l, and • Confusion matrix [17]: a matrix C to evaluate the accuracy of a classification, where C i,j is the ratio of the number of observations in group i and predicted to be in group j over the number of total observations in group i.
One overall list of models and their evaluation results are shown in Table 4. The corresponding training time for them on Google Colab is also presented in the Table 5, where running time may vary significantly due to computation resource fluctuation. More details about model implements and experiment results are provided in the following subsections.

Machine Learning Models
In the first part of experiment, we apply four machine learning models, i.e. Naive Bayes, Logistic Regression, Random Forest, and Linear Support Vector Machine (SVM), for the Yelp review rating prediction task. Classifiers from scikit-learn [12] are used.

Naive Bayes
The first applied machine learning model is Naive Bayes classifier [8] for multinomial models (MultinomialNB). The accuracy and weighted F 1 score on the testing set are shown in the Table 4, and the confusion matrix is shown in the Figure 3. Naive Bayes model is convenient to be implemented with short running time, which can be viewed as the benchmark for further models.

Logistic Regression
The second applied machine learning model is Logistic Regression [8] (LogisticRegression) with L 2 penalty and LBFGS solver. The accuracy and weighted F 1 score on the testing set are shown in the Table 4, and the confusion matrix is shown in the Figure 4. Logistic Regression model outperforms the Naive Bayes with 64% accuracy, and this result matches reported one [5].

Random Forest
The third applied machine learning model is Random Forest [8] (RandomForestClassifier) with 500 estimators, minimum samples leaf 10, and Gini impurity. The accuracy and weighted F 1 score on the testing set are shown in the Table 4 with the confusion matrix in the Figure 5. The prediction is skew towards two polarities. This is because both 1 and 2 stars reviews have negative sentiment words, and 4 and 5 stars reviews share positive ones. Decision tree

Linear Support Vector Machine
The fourth applied machine learning model is Linear SVM [8] (SGDClassifier, linear classifier with stochastic gradient descent training) with hinge loss and L 2 penalty, where StandardScaler is used for feature preprocessing. The accuracy and weighted F 1 score on the testing set are shown in the Table 4, and the confusion matrix is shown in the Figure 6. Similar polarity skewness as Random Forest can also be found in this linear SVM model, although the accuracy is higher than the Naive Bayes.

Transformer-Based Models
In recent years, transformer-based models [18] become the important Natural Language Processing (NLP) landscapes, and they outperforms many other state-of-the-art In the second part of experiment, four transformer-based models are used on our Yelp review rating prediction task, including BERT [4], Distil-BERT [16], RoBERTa [10], and XLNet [20]. One NLP Python package simpletransformers [14] based on transformers [19] and pytorch [11] is used in this section.

BERT
The first applied transformer-based model is BERT. BERT (Bidirectional Encoder Representations) [4] is a language representation model, where deep bidirectional representation from unlabeled text is pre-trained on both left and right context for masked language model and next sentence prediction tasks. Such pre-trained BERT model can used for wide tasks after being fined-tuned with just one additional output layer.
Two BERT architectures are available: one smaller model called base BERT with 110M parameters (12 transformer blocks, 768 hidden size, 12 self-attention heads), and another larger model called large BERT with 340M parameters (24 transformer blocks, 1024 hidden size, 16 self-attention heads). Both of them can be uncased and cased, depending on the pretraining English text. We use bert-base-uncased, bert-base-cased, and bert-large-cased, and their results on the validation set are shown in the Table 6. It can be found that the cased, large models outperform the uncased, based ones. This conforms to the usual expectations, since upper-case words could have different meanings than lower-case ones, and lager model has more parameters to fit the data. However, large BERT model only gives 0.31% increase in accuracy with double running time. To keep a reasonable running time on Google Colab, we will use base architectures for other transformer-based models. For the transformer-based model, one of hyperparameters to be determined is the maximum sequence level. Base BERT models pretrained on the cased English text with maximum sequence level 32, 64, and 128 are chosen here for simplicity and representativeness. The results on the validation set are show in the Table 7, where one can find the larger maximum sequence level is, the higher accuracy, since longer sequences can contain more useful information for classification. Although shorter sequence length will give worse results, it is still meaningful when memory resources are limited and faster computation is needed. Since the maximum sequence level with 256 will be out of memory on Google Colab, we will use 128 as the maximum sequence level for other transformer-based models. In summary, we apply BERT models with following settings: • Architecture: base model pretrained on uncased (lower-case) or cased English text, • Maximum sequence length: 128, • Training batch size: 16, • Number of training epoch: 1.
The accuracy and weighted F 1 scores on the testing set are shown in the Table 4. The corresponding confusion matrices are shown in the Figure 7 and 8. By comparing the results from base BERT models with the Logistic Regression, we can find the following outcomes: • The overall accuracy increases from 64% to 69%.
• The weighted F 1 score increases from 0.64 to 0.70.
• The BERT models have good abilities to distinct categories between 2-4 stars, which gives higher scores on diagonals of confusion matrices.

DistilBERT
The second applied transformer-based model is Distil-BERT. DistilBERT (Distilled BERT) [16] is a distilled version of BERT, where 97% language understanding capabil-ities of BERT are retained with 60% faster performance and 40% smaller model size.
The accuracy and weighted F 1 scores on the testing set are shown in the Table 4 with corresponding confusion matrices in the Figure 10 and 11. Compared with BERT, DistilBERT model has lower accuracy in around 0.5% with almost double speed up. When the computation resources are limited, DistilBERT will be an appealing choice.

RoBERTa
The third applied transformer-based model is RoBERTa. RoBERTa (Robustly optimized BERT approach) [10] improves pretraining procedure and achieves state-of-the-art results on several NLP tasks.
For this project, we use pretrained roberta-base architecture. The accuracy and weighted F 1 score on the testing set are shown in the Table 4. The confusion matrix is also shown in the Figure 12. Higher scores are achieved as expected compared with the BERT model. A base RoBERTa even outperforms a large BERT model with comparable running time. This suggests the RoBERTa model will be a good choice for higher prediction metrics with sufficient computation computational resources.

XLNet
The last applied transformer-based model is XLNet. XLNet [20] is a generalized auto-regressive pre-training method improved from BERT. In XLNet, all possible permutations for tokens are predicted, while only masked tokens in BERT. Also, ideas from Transformer-XL are integrated into XLNet. These allows XLNet outperforming BERT on 20 NLP tasks. In this classification project, we use pretrained xlnet-base-cased architecture. The accuracy and weighted F 1 score on the testing set are shown in the Table  4 with the confusion matrix in the Figure 13. XLNet model gives the best performance over all models we tried, also the longest running time. The 70.44% accuracy is achieved on the testing set.

Conclusions
In this paper, we predicted ratings from Yelp review texts. Yelp Open Dataset was used. The imbalanced data distribution was presented, and a balanced training dataset was built. Four machine learning models including Naive Bayes, Logistic Regression, Random Forest, and Linear Support Vector Machine were used based on numerical features from tf-idf vectorizer. Four transformer-based models including BERT, DistilBERT, RoBERTa, and XLNet were also trained and tested on the textual data. Comparisons between models and hyperparameters were done, and 64% accuracy score for the machine learning model and 70% accuracy score for the transformer-based one were achieved on the testing set.
Transformer-based models were summarized and experimented. Cased, large BERT models were found giving better performances than the uncased, base ones. DistilBERT has a faster computation speed with a bit lower metrics, while RoBERTa and XLNet give higher evaluation metrics with more computational resources required.
We hope our work could give some inspirations and insights for further work in Yelp review rating prediction based on machine learning and deep learning models.