{'fraction_non_alphanumeric': 0.027567195037904894, 'fraction_numerical': 0.003962784286698828, 'mean_word_length': 5.529246344206974, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Overview of Collaborative Filtering Recommendation Algorithms

In this paper, the key technologies of collaborative filtering recommendation algorithm and the bottleneck in its development are summarized, the problems of different technologies are analyzed. Besides, the application prospect of collaborative filtering technology is prospected.


Introduction
With the rapid popularization of e-commerce, social network, MOOC, P2P and cloud computing, the scale of Internet information resources is rapidly expanding [1]. Faced with the mass data, how to use intelligent technology to discover products and services that meet individual interests and preferences has become an important research topic [2]. Collaborative filtering recommendation is one of the most popular technology in current recommendation systems [3]. Collaborative filtering algorithm analyses user preferences and finds user sets with similar interests to designated users. According to the viewpoint of similar users in user set, the recommendation list of target users is deduced, and then user preferences are predicted. Collaborative filtering recommendation is based on the assumption that if users have similar ratings on some items, indicating that they have similar preferences, they also have similar ratings on other items.
In the industry, recommendation systems play an important role in various highly rated website platforms and e-commerce systems, such as Google News, Amazon.com, Netflix, Last.fm, YouTube, Spotify, Twitter, LinkedIn, and Facebook.
In academia, many domestic and foreign researchers and research institutions have carried out on innovative development and in-depth research in the field of recommendation systems. At present, recommendation system has developed into an independent and popular research discipline. In 2007, the Computer Association (acm) began to host the International Conference on Recommendation Systems. In many international conferences in the fields of human-computer interaction, information retrieval, data mining and machine learning, (such as ACM SIGIR, ACM SIGMOD, ACM SIGKDD, UMAP, IUI, WWW, ICML, ICDM, etc.), the number of academic papers on recommendation system is also increasing. Some high-level academic journals have also set up special journals on recommendation systems to collect and publish the latest research results in this field, such as:

Main Challenges of Collaborative Filtering Recommendation System
Recommendation system has prominent performance in stimulating users'purchase enthusiasm and promoting enterprise revenue, which has attracted more and more attention of enterprises. However, with the rapid growth of Internet data and the changing application scenarios, the demand for applications in various fields is also increasing. There are still many new problems and challenges to be solved urgently in recommendation system, such as Data Sparsity [4], Cold Start Problem, scalability [5], accuracy and diversity [6][7], Top attack problem [8][9], context awareness problem [10], privacy protection problem [11], etc.

Data Sparsity
The problem of data sparsity is caused by insufficient or even missing useful information. With the rapid development of various application platforms and systems in the Internet, the number of users, especially the number and types of projects and services, has increased explosively. At the same time, in the face of massive services and projects, the rating information or useful historical information is relatively limited, and the useful historical information matrix between users and projects is often very sparse, even with the increase of users and projects, it will show super high-dimensional and uneven distribution. The cold start problem in recommendation system is equivalent to the extreme case of data sparsity problem. Because of the lack of user feedback data, the system cannot predict based on historical information or give a valid recommendation [12]. Therefore, In this case of high sparsity, it can neither accurately describe users'interest preferences nor accurately calculate the similarity of their interests, which will lead to the decline of recommendation quality and accuracy.
In order to alleviate this problem, the main solutions are: converting other auxiliary information into scoring information, so that relevant recommendations can be given to each user based on available data, such as recommendation algorithm based on demographic information, recommendation algorithm based on social relations and hybrid recommendation algorithm [13][14][15][16]; fill in the vacancy value of the original sparse matrix [17]; use dimensionality reduction technology to extract features to make the data mapped to low-dimensional space relatively dense [18]; consider the relevance of second-order or higher-order global information [19][20]; combine with content-based recommendation methods [21]; improve the accuracy of similarity measurement methods [22][23].

Cold Start Problem
New users and new objects are cold start problems. In recommendation system, especially in collaborative filtering system, new objects must wait for a period of time before users can view and evaluate clicks, scores, comments and so on. It is impossible to analyze and recommend this object before the evaluation reaches a certain number. Different from the new user problem, this kind of problem is generally considered to use the combination recommendation method to deal with. Paper [24][25]: Using object entropy, popularity and user personality attributes to improve the effect.

Scalability
For large-scale Internet sites, recommendation systems usually need to face more than one million users and commodities, and at the same time need to search the similarity in the whole user space in real time, so as to complete the recommendation for target users. Therefore, when the number of users and products in recommendation system increases rapidly, the traditional recommendation system faces severe challenges in real time and scalability of the algorithm. Many scholars'research results show that the recommendation accuracy and real-time performance of the recommendation system are usually a contradiction, it is difficult for the recommendation system to meet the real-time requirements while ensuring high-quality recommendation accuracy. Therefore, on the premise of guaranteeing the best recommendation quality, how to ensure the real-time and scalability of the recommendation system is one of the challenges faced by the recommendation system.
In a word, the problems mentioned above restrict the further development and application of recommendation system, which is a difficult problem and challenge that needs to be paid close attention to.

Comparison of Nearest Neighbor Selection Methods
Computing the similarity of users or items is an important part of collaborative filtering recommendation algorithm. In collaborative filtering recommendation technology, there are several commonly used methods to calculate the similarity:

Pearson Correlation Similarity
Assuming that the item set scored by user(i) and user(j) is I ij , the similarity sim(i,j) can be obtained by Pearson correlation.
, and , represent the score of user(i) and user(j) on item c, and represent the average score of user(i) and user(j) on all items.
Because Pearson correlation coefficient is obtained by linear regression formula, it is necessary to satisfy the assumptions of linear relationship between the data and that the residual is independent of each other and the mean value is zero. When these conditions are not satisfied, the accuracy of calculation will be reduced.

Jaccard coefficient
Jaccard coefficient is used to measure the degree of overlap of binary data. It is defined as follows: In e-commerce, Jaccard coefficients are usually used to compare shopping cart data of different users, and this similarity measure, which is only applicable to binary type, also limits its further application in recommendation system.

Cosine Similarity
Cosine similarity is often used in the field of information retrieval. The similarity is measured by calculating the angle between document vectors.
In practice, different scales of user ratings are inconsistent, some users tend to score high, while some users tend to score low, then cosine similarity can not accurately measure the similarity between users.

Modified Cosine Similarity
The common cosine similarity measurement method does not consider that different users may have different scoring scales. Therefore, the modified cosine similarity improves the defect and eliminates the difference of rating scales by considering the average of user ratings.

Average Weighting Strategy
A basic assumption of collaborative filtering algorithm is that users with similar preferences will give similar ratings for the same project. Therefore, after the target user's nearest neighbor set is generated, it can predict the target user's score for the non-scored items according to the user's score in the nearest neighbor set.
At present, most collaborative filtering recommendation systems use average weighting strategy to generate recommendation.
, represents the predictive score of user(u) for item(i), and KNB represents the nearest neighbor set of target user(u).

Top-N Recommendation Strategy
Top-N recommendation strategy is to calculate the weighted average of user(i)'s interest in different items in the "nearest neighbor" set, and take N items which rank first and do not belong to Ii (Ii represents the item set of user(i)'score) as the Top-N recommendation set.

Comparisons of recommended quality assessment methods
The advantages and disadvantages of a recommendation system are measured by its prediction results. At present, the evaluation strategies commonly used in collaborative filtering recommendation algorithms are as follows.

Average Absolute Error (MAE)
Average absolute error is the most widely used evaluation method in the recommendation system. It is obtained by calculating the absolute error between the predicted value and the actual value.
Variable n is the total score, represents user(i)'s predictive score for item j, represents user(i)'s actual score for item(j). The smaller the MAE value, the higher the recommendation accuracy.

Root Mean Square Error (RMSE)
RMSE, also known as standard square deviation, reflects the discreteness of scoring data.
The smaller the RMSE value, the higher the recommendation accuracy.

Recall
The recall rate is used to reflect the recommendation rate of the project to be recommended.
The test represents the number of items in the test data set, the top-N represents the N items recommended by the system to users. The greater the Recall value, the greater the probability of recommendation.

Conclusion
This paper mainly introduces the commonly used nearest neighbor selection algorithm, recommendation algorithm and evaluation strategy of recommendation algorithm when using collaborative filtering algorithm for recommendation. Collaborative filtering recommendation algorithm mainly has the problems of data sparsity, cold start and robustness, as well as the recommendation efficiency in large data environment. Researchers have proposed a variety of solutions, the most common one is to introduce methods from other fields, and the interdisciplinary research of collaborative filtering has been further developed. With the rapid growth of information on the Internet, improving collaborative filtering recommendation based on large data is a formidable challenge.