{'fraction_non_alphanumeric': 0.028252572187188848, 'fraction_numerical': 0.007923996017258546, 'mean_word_length': 4.7188612099644125, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
House Price Prediction Based on Machine Learning: A Case of King County

This paper focuses on formulating a feasible method for house price prediction. A dataset containing features and house price of King County in the US is used. During the data preprocessing, extreme values are winsorized and highly correlated features are removed. Eight models including Catboost, lightGBM and XGBoost serve as candidate models. They are evaluated by several evaluation indicators, including rooted mean square error, R-squared score, adjusted R-squared score and K-fold cross validation score. The model that has low RMSE, achieves a high R-squared score and adjusted R-squared score, especially in the test set, and acquires a high score in cross validation is considered a better model. This paper finds out that Catboost performs the best among all models and can be used for house price prediction. Location, living space and condition of the house are the most important features influencing house price. After comparison and contrast with other papers, it is attested that findings in this paper conform to real life. This paper formulates a model that fits better than preceding studies for house price prediction and makes necessary supplement to the exploration of features that influence house price from a microscope.


INTRODUCTION
The trend of house prices is always a controversial topic as its fluctuation will pose a huge effect on the entire economy. The rise in house price means growth in nonfinancial assets which ultimately increase personal wealth, stimulating household consumption and boosting the economy; however, a decrease in house price limits an individual's borrowing capacity, crowding out investments due to the evaporation in the value of collaterals [1]. The shock in the global economy caused by the 2008 housing bubble perfectly explains the importance of a stable and measurable house price. The turmoil in house prices causes an unexpected rise in real long-term interest rates, bankruptcy in financial institutions and global economic depression [2]. Although it is hard to control the house price, it is possible to predict it.
Many scholars have conducted research on this issue. For instance, Hirata et al. have used time-series models to determine that house prices have become more synchronized over time and the FAVAR model to find out that global interest rate shock has the most considerable influence on global house price, especially in the US [3]. Shishir Mathur has provided insight from a micro perspective in his report, stating that quality and size are two factors, contributing to house price [4]. In his opinion, this assumption can be explained through the perceived value of the house. The property assessors will evaluate the size and quality of the house during value assessment processes for house reselling, which will determine the value of the house. Property developers also take houses' quality and size into consideration while they initially design the project and pricing for the property. A bigger size and better quality will bring a higher perceived value to both assessors, developers, and buyers. In Shishir Mathur's report, he also mentioned another contributorthe level of maintenance. With the increased investment in refurbishing before offering for sale, the house owners will expect a higher dealing price due to their value addition through the maintenance. Although prior researches make a valid analysis, they failed to discuss the simultaneous effect of those factors. This paper goes beyond previous economic analysis and uses machine learning to explore the country-wide house price. This paper also assumes that the two factors, size and quality mentioned by Shishir Mathur, will affect the house price, but this paper will use Machine learning algorithms to prove the relationship. Other than these two factors, some other factors on house price, including location, size, and overall structure of the house and grading from the agency will also be evaluated. Machine learning's most obvious advantages are that it can automatically solve a wide range of problems and efficiently handle big datasets [5]. These two benefits allow us to prove the assumptions through analyzing a huge amount of historical data and taking multiple factors into consideration to present a comprehensive model efficiently.
The research studies the house price in King County, US, during a 2-year period from 2014 to 2015. According to the data gathered by Washington Government, King County has the highest estimated population of 2,052,800 in 2015 among all counties in Washington [6]. With a higher population, King County has more potential house buyers and higher house demands; thus, house price data in King County will be more complete and more precise. This paper employs different technical models, including Catboost, LightGBM, XGBoost, Random Forest and regressions to identify the important influencer of the house price. The best model will be selected through training and testing, which will allow us to have the most accurate result. The results will conclude important micro features in determining the house price, including the location, size, and gradings. This will provide a guide for future house price prediction to not only consider the macro effects but also think about the micro factors.
The rest of the paper is organized as follows: section 2 and section 3 introduce the source of the data and the methodology used in the evaluation. The fourth section discusses the evaluation and the results. Eventually, the conclusions are in section 5.

DESCRIPTION OF THE DATASET
The dataset used to predict the sales price of houses in King County comes from Kaggle. It includes 21613 observations of 20 house features and one house price column for homes sold between May 2014 and May 2015.
Among the 20 features, eight of them are the continuous numerical variables, that describe the area dimensions in measurements and the geographical location of the house. These continuous variables provide a basic view of the overall structure and information of the house. The rest of the attributes are discrete variables, which provide some more detailed information on components of the house. Most of them quantify the number of items in the house, for instance, the number of bedrooms, bathrooms, waterfront, and floor. Some others indicate the background of the house, such as year of building, year of innovation and previous selling price and date. One thing that should be mentioned is that values in the attribute, "yr_renovated", will be replaced by the difference between the year of renovation and the year sold out. Additionally, there are two evaluation scores: "Grade" and "Condition". These two attributes grade the overall condition of the house based on different scales and standards [7].

DATA PREPROCESSING
A classic phrase in computing says "garbage in, garbage out" [8]. In another word, "good" data is the origins of high-quality analysis and project design. Foxwell concluded four causes of data error: creation and pre-collection errors, collection errors, post-collection and analysis errors, and recording errors [9]. During this stage, this paper will focus on addressing collection errors of the dataset, especially the two frequently mentioned collection errors: missing variables and outliers. Since the counts of each feature are equal to the total number of observations, the data does not have any missing variables. However, there is no outliers in the data. Comparing the price at 75% quantile ($645000) with the maximum number ($7700000), feature "price" should have some outliers ( Figure 1). After further analyzing the distribution of "price", the right-skewed distribution with a fat tail confirms the outliers. Thus, the price which are greater than 99th percentile number will be replaced with 99th percentile numbers. The same method will be applied in the other numerical features. After adjusting the dataset into a "good" version, the paper further explores the implicit meaning of the raw data. Firstly, the distribution of the features illustrates that most of the house prices collected in the data come from houses with 2 to 5 bedrooms (  Figure 5). Houses with this structure will be more attractive in the real estate market. Secondly, most of them have the overall condition of a house is at the 3rd level on a scale of 1 to 5 and the overall grade given to the housing unit, based on King County grading system is at level 7 on a scale of 1 to 11. This demonstrates that the property in the data is mid-level houses.    The observation from the heatmap in Figure 6, demonstrates that most of the data concentrated in the west of King County, especially in Seattle. Data are rare in cities located in the east of the county, such as Snoqualmie and Skykomish. This is because most of the area in east King County are covered by forest.

FEATURE SELECTION
According to the correlation matrix (Figure 7), "sqft_living15" has a high correlation with "bathrooms" (0.57), "sqft_living" (0.76), "grade" (0.71), and "sqft_above" (0.73). Therefore, this feature will be dropped to avoid multi-correlation and increase the accuracy of the result. Additionally, the feature, "sqft_lot15" which has a similar meaning to "sqft_living15" will also be dropped. Lastly, because "id" does not have any noticeable relationship with house price, it will be deleted.

METHODOLOGIES
This paper tests eight regression models, as implemented in the Sci-kit learn, XGBoost, Catboost and LightGBM package of Python. Those models include multiple linear regression, polynomial regression, lasso regression, ridge regression, random forest regression, XGBoost regression, LightGBM regression as well as Catboost regression.

MULTIPLE LINEAR REGRESSION AND POLYNOMIAL REGRESSION
In multiple linear regression, the output is subject to 1 , 2 , … . It is determined when 0 , 1 , … , are chosen [10]. It can be represented as: Similarly, the polynomial regression model can be written as: where i denotes the degree of independent variables [11].

RIDGE AND LASSO REGRESSION
In ridge regression, the goal is to optimize the following program: where is a parameter used to balance the regularization factor and the error. The second factor is 2 regularization, which avoids overfitting [12].
Similarly, in lasso regression, the goal is also to optimize a program.

Advances in Economics, Business and Management Research, volume 648
The second factor is 1 regularization.

RANDOM FOREST REGRESSION
Random forest searches for the best feature among a random set of features. It trains the model for T rounds. The best feature in each random subset is used to split the node and the combination of them generates the strong learner F(x).

XGBOOST AND LIGHTGBM REGRESSION
XGBoost is a method, originated from gradient boosting decision tree (GBDT). Its objective is to find the function f(x) to fit the residual error in the last node so that the loss function is reduced to the minimum [13]. LightGBM is another method based on GBDT. It adopts the histogram algorithm, which reduces the computational cost. It also adopts leaf-wise tree growth. Every time the tree grows, it splits from the node that performs the best and iterates this process. What's more, it controls the maximum depth to avoid overfitting [14].

CATBOOST REGRESSION
The model that outperforms other ones is Catboost, and it is worth diving deeper into its algorithm. Catboost adopts the gradient boosting procedure. Ft is built iteratively in a greedy fashion, representing a sequence of approximations. Ft is obtained from the following equation: where is the step size, and ℎ is chosen to minimize the following loss function.
The solution to the problem is usually obtained by functional gradient descent. The gradient step ℎ is selected in the way that ℎ → − ( , ) , where ( , ) = ( , ) | = −1 ( ) .Often, the method utilized for the approximation is the least-squares approximation.
Catboost adopts a decision tree as its base predictor. The decision tree divides the feature space into disjoint regions according to the values of some splitting attributes a. Splitting attributes are usually binary ones that identify that some features exceeds some threshold t. This can be written as = { > } , where is either a numerical or binary feature. The final node of the tree serves as the estimate of the response y. A decision tree, therefore, can be written as: where is the disjoint regions corresponding to the leaves of the tree [15].
Catboost makes use of a strategy named Ordered TS (Target Statistics) in the prevention of prediction shifts. To realize this strategy, an artificial "time", i.e., a random permutation of the training examples, is introduced. Then, we take = { : ( ) < ( )} as the training example and = for a test one, where is the dataset. This strategy not only uses all the training data for the learning model but also satisfies the following property: where ( , ) is the k-th training example.

EVALUATION INDICATORS
After preprocessing the data, we fit the data into the models and acquired the outcome. With the purpose of evaluating the models, we picked several statistical indicators.
The first indicator is the Root Mean Square Error (RMSE). It can be utilized to measure the precision of a regression model. The way that RMSE is calculated is written as: where m is the number of instances in the dataset, ( ) is a vector of all feature values of the ith instance, ( ) is the target value for each instance, is a matrix containing all feature values and h is the system's prediction function.
The second indicator that we picked is R-squared. The greater the value of R-squared, the better the model fits. The maximum value of R-squared is 1. R-squared is calculated in the following way: where 2 is the value of R-squared, is the true value of target observation, ̂ is the predicted value, and ̅ is the mean value for the target vector.
Third, we selected the value of adjusted Rsquared(adjusted 2 ) because there is a problem with 2 : when the total number of features increases, the 2 will also increase, regardless of whether the variable is indeed closely related to the target variable. Adjusted 2 can be denoted as: where p is the number of variables and n is the number of instances. What's more, we adopted K-fold crossvalidation, as is demonstrated in Figure 8. K-fold is a vivid description of dividing the whole dataset into K parts with the same amount of data. During the process of K-fold cross-validation, firstly, as the name K-fold indicates, we first divide the dataset into K equal parts. After that we let one part serve as the test set, and the remaining parts serve as the training set. Test set is used to conduct model evaluation and see whether the model predicts well, while training set is used to train the model to fit data. Specifically, to begin with, the first part serves as the test set, and the remaining K-1 parts serve as the training set. Then, the second part is used as the test set, and then the third, etc. It iterates this procedure K times until the Kth part has served as the test set.

MODEL EVALUATION AND SELECTION
After determining the candidate models as well as the evaluation indicators, we processed the data with Python. During the evaluation process, 2 and adjusted 2 are calculated not only for the training set but also for the test set, helping us to see clearly how the models are performing respectively in the two sets. K-fold crossvalidation is conducted on the whole dataset to holistically assess how well the model is performing. The results are arranged in descending order by K-fold crossvalidation score, where K =5 in the experiment. Table 1 illustrates the results. Notes: alpha stands for regularization parameter, degree stands for the highest degree of polynomial regression, and the name in the column 'model' stands for its kind of regression, for example, Catboost stands for Catboost regression.
From Table 1, it is not difficult to conclude that Catboost Regressor performs the best among all models. It has an RMSE of 95163.23 and becomes the only model that has an RMSE of less than 100,000. When it comes to the R 2 score, adjusted R 2 score, as well as the 5-Fold Cross Validation score, Catboost stands out from the candidate models as well. Catboost demonstrates a great capability of precise prediction and does not show any tendency of overfitting, therefore, there is no doubt that Catboost is selected as the final model used to predict house prices.
The hyperparameters in the model are set by default. Here, we discuss some of the hyperparameters that are most used. In the model, the 'iteration', which means the largest number of trees, is set to be 1000. 'Learning rate' is set to be 0.03. 'Depth' means the maximum depth of the tree, which is 6. 'Class_weights' determines the Advances in Economics, Business and Management Research, volume 648 weight of each category, highly useful in hierarchical training with unbalanced data, is set to be None.
It is worth noting that the model that obtains the highest average R 2 score of the training sets in each iteration is Random Forest Regressor, which achieves an R 2 score of 0.984. However, when it comes to the average R 2 score of the training sets in each iteration, its performance is not that ideal. The average R 2 score drops to only 0.885. It is suspected reasonably that there is a slight problem of overfitting with Random Forest Regressor.

IMPORTANT FEATURES FOR DETERMINING THE HOUSE PRICE
This section will explore what features bring the most influence to the outcome of the model. The graph in Figure 9 shows feature importance generated through Catboost.

Figure 9
Feature importance graph of Catboost regressor model. Some features that get a high score in feature importance are worth discussing. The first one is location. Note that although latitude ranks the first among all features, longitude also gets a high score and is supposed to be taken into consideration. After all, the combination of latitude and longitude represents the location of the house, thus influencing the house price. This highly conforms to real life. Location always serves as the determining factor for house price. An example for this argument prevails, for instance, a place that is convenient with public transportation is usually sold for a higher price. Likewise, a place near parks or lakes is priced highly for its surroundings. The second important factor is the area of living space. It is not surprising for 'sqft_living' to come to this place because when houses are sold, they are priced a certain amount of money per square meter or square foot. Therefore, the larger the house, the greater amount of money customers need to pay. The third important factor is the grade of the house. It is reasonable that a house in good condition will be more attractive to consumers, resulting in a higher price for sale. The feature importance outcome in this case, in general, is highly compatible with our consensus. House sellers ought to pay more attention to these features to gain more revenue and attract customers.

FURTHER DISCUSSION
This paper mainly focuses on the prediction of house prices from a scope that is comparatively micro. Properties of the houses are utilized to determine how much the houses are priced and what the important factors are in influencing house prices. A similar approach to searching for the determinant factors of house prices has also been used in a good sum of papers. A comparison and contrast of the findings of papers will be conducted in this section.
According to Mathur, who also conducted a survey of house prices in King County of the United States, the size and quality of a certain house matter the most for determining the house price. Bigger size and better quality will bring about a higher estimated value for assessors. Such a finding highly conforms to our outcome, where living space and grade rank among the top three. He also finds that a higher level of maintenance will make the house appreciate. Scholars studying other areas also contribute to this topic. Zakaria and Fatine conducted research on determinants of real estate's price in Morocco. They find out that two factors most significantly determine the house price, which is a surface area as well as the location of the real estate [16]. These two factors rank the second and the first, respectively, in our finding. Selim did research on figuring out the determinants of house prices in Turkey. Taking even more properties into account, he concludes that the condition of the water system, whether the house has a swimming pool, and the type of the house (what material the house is made of) are the most important factors [17]. These factors seem to obviate the previous findings. However, if inspected carefully, these factors are, to some extent, related to the grade of a house. Besides, he mentions that the number of rooms and the locational characteristics is also important. These factors are compatible with our findings in this paper.
All literature mentioned above solves the problem of house price prediction and important factor determination from a microscope. Extant literature effectively attests to the validity of our paper's findings. Though there exist some slight differences, the general outcome is quite similar. House location, the space for living, as well as the condition of the house, are indeed among the most essential features from a microscope to determine how a certain accommodation will be priced.

CONCLUSIONS
In this paper, the issue of house price prediction is explored using a case from King County in the United States. In order to eliminate the problems that exist in the original dataset, this paper not only winsorizes the extreme values in numerical features like 'price', but also calculates the correlation coefficient and removes the highly correlated features including 'sqft_living15' and 'sqft_lot15', to assure with a precise prediction. Then, several models are utilized to fit the data. They are assessed with a variety of evaluation indicators including RMSE, R 2 score, adjusted R 2 score and cross-validation score. Among the models, Catboost outperforms all the other models and becomes the selected model because it derives the highest R 2 score and adjusted R 2 score in the test set and ranks the first in cross-validation score. The final model and corresponding essential factors are subsequently derived through Python coding. Comparison among related literature is also conducted to complete a further discussion of the topic.
From the research, we obtain the following conclusions. First, Catboost serves as the best model for our house price prediction. It not only gets the highest score in a model assessment and makes a sensible prediction, but also avoids overfitting. Second, the most important factors in the microscope that influence the house prices are location, living space and the condition of the house. Such a finding highly conforms to our common sense.
The innovations of this essay are summarized as follows. First and foremost, this essay adopts Catboost to predict house prices. This approach achieves better prediction precision compared to extant research papers on the same issue. compared to extant research papers on the same issue. In addition, this essay focuses on the house price prediction from a microscope rather than macro scope which is used by more scholars. This brings about an essential supplement to research on the house price prediction.
Despite the merits above, this essay still bears some slight drawbacks. First, this paper does not cover the macroeconomic factors. If they were taken into consideration, the results might be closer to real-life situations. Besides, this paper conducts a case study of King County of the US. However, for other areas that are not similar to King County, additional study is probably needed.