{'fraction_non_alphanumeric': 0.03292290039277991, 'fraction_numerical': 0.005957897524162584, 'mean_word_length': 5.005830903790088, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Novel Privacy-Preserved Recommender System Framework based on Federated Learning

Recommender System (RS) is currently an effective way to solve information overload. To meet users' next click behavior, RS needs to collect users' personal information and behavior to achieve a comprehensive and profound user preference perception. However, these centrally collected data are privacy-sensitive, and any leakage may cause severe problems to both users and service providers. This paper proposed a novel privacy-preserved recommender system framework (PPRSF), through the application of federated learning paradigm, to enable the recommendation algorithm to be trained and carry out inference without centrally collecting users' private data. The PPRSF not only able to reduces the privacy leakage risk, satisfies legal and regulatory requirements but also allows various recommendation algorithms to be applied.


Introduction
With the explosive development of online services, products, and information resources, the phenomenon of "information overload" is becoming more serious, which brings a significant information burden to people. Recommender Systems (RS) is currently an effective way to solve information overload and realize personalized information services. Currently, RS has played a core role in many domains, such as e-commerce (Alibaba, Amazon), video service (Netflix, YouTube, and TikTok), online-music services (Pandora, Yahoo), Advertising (Google, Facebook), both the user and service providers benefit from the personalized RS. However, the current RS exists many potential risks, in which privacy leakage is one of the top concerns [1]. The ultimate goal of RS is to understand user preference better than users themselves [2] [3]. RS needs to collect users' personal information and behavior to achieve a comprehensive and profound user preference perception to meet users' next click behavior. The prediction quality varies depending on the scale, diversity, and timeliness of the data collected. However, these data quality factors are also positively related to privacy risks and the degree of harm caused by leaking these data. This is an unavoidable "privacy-personalization trade-off problem [4] [5]"to the current RS. The RS's privacy risk is mainly due to internal and external improper operations and attacks on the centrally collected and stored user information. As illustrated in Figure 1, internal privacy threats come from a curious or malicious service provider. Driven by commercial interests, service providers may violate privacy contracts and collect user data without authorization or share data with third parties. Internal staff may also use access permissions to spy on users' privacy. External privacy threats mainly come from hacker attacks. Due to the lack of adequate data security protection, data is maliciously stolen. Sometimes there will be unconscious data leakages, such as data surveillance by law enforcement agencies or publicly anonymous data being de-anonymized (de-anonymization attack [6]) by a third party.
In May 2015, eBay was attacked by hackers, resulting in the disclosure of 145 million user accounts, including user names, addresses, data-of-birth, and account password. In March 2018, Facebook leaked tens of millions of users' personal information. Many Internet companies have similar incidents, leading to large-scale user privacy data leakage, triggering user trust crisis. Consequently, people raised their attention to protect their data privacy and security and avoid their private information being collected by Internet applications. Relevant data security and privacy protection laws and regulations have been promulgated as well. The US proposed the "California Consumer Privacy Act " (CCPA) 1 , the EU published "General Data Protection Regulation" (GDPR) 2 , Singapore published "Personal Data Protection Act" (PDPA) 3 , and "Data Security Management Measures (Draft for Comment)" published by China. These regulations rely on a fundamental principle: It is up to individuals, groups, or organizations to decide when, how, and to what extent to communicate information about them to others. By Alan F. Westin [7]. These regulations have made user data collection face many obstacles without users' full authorization. In this context, the user's local device firmly locks the critical data, forming the isolated data islands that are unable to aggregate, cause the RS to no longer personalized, and user experience also declines.
The emergence of Federated Learning (FL) connects isolated data under the premise of ensuring data security and user privacy, thereby providing the possibility to train accurate and robust machine learning models. McMahan et al. [8] first introduced the term FL in 2016 and applied it to update smart-phones' voice recognition and text entry models. The Google team utilize the user's mobile phone to train the local user interaction data, which can continuously optimize the user's local language prediction model, and through the "Federated Averaging" algorithm (FedAvg), by averaging all stochastic gradient descent (SGD) result on user's terminal with a central server, a globally optimized model is obtained. Currently, the study of FL has been carried out in many application fields [9][10] [11] related to privacy protection and data security.
In this paper, we introduced the FL paradigm into the RS and proposed a novel privacy-preserved RS framework based on FL (PPRSF) that enables the training and online inference under the premise of preserving user privacy, satisfying legal and regulatory requirements.

Related Work
In this section, we review relevant studies of privacy-preserved RS and federated learning to position our research in relation to existing studies.

Privacy-preserved RS
In response to the privacy threats, privacy-preserved RS use specially designed architectures or algorithms as countermeasures.
Architecture-based Solutions Architecture-based solutions are designed to minimize the threat of data leakage. For example, using distributed storage of user data can effectively reduce the damage caused by a single data source exposure, or distributed the recommendation process to increase the difficulty of unauthorized access to data. Based on distributed data storage, Heitmann et al. [12] proposed a candidate architecture, in which users hold local configuration data, and the users decide which data can be disclosed to which service provider, only holding specific certificates the application can access this part of configuration data through API and use it for recommendation calculation. This method allows users to make data disclosure decisions, but it cannot avoid collecting open data by curious service providers, while users often choose to open data to obtain better services. Based on the distributed recommendation process, some studies have proposed to design and implement the recommendation process in a pure P2P system [13][14] [15], compare local user data with another user for similarity to obtain possible recommendation results, remove the role of the central server to avoid individuals Centralized storage of information. However, such methods may still leak user data to other users and impose requirements on local devices' computing power.

Algorithms-based Solutions
The algorithm-based solution makes certain modifications to the original data so that even if the data or model output is obtained by a third party, sensitive information will not be exposed. The main methods including differential privacy (DP) and encryption algorithms.
• DP is a privacy model based on data fuzzification, which can alleviate the risk of inferring users' private data through the output of the recommendation system. The core idea of DP is that when the enemy tries to query individual information from the database, the data is confused by noise, so that the enemy cannot distinguish the sensitivity of the individual from the result. McSherry and Mironov [16] first applied DP to collaborative filtering, using the Laplace mechanism to generate corresponding noise on the input score, which can shield the impact of the change of a single record on the final result. Follow-up research includes [17][18] However, due to noise interference, DP cannot guarantee privacy and the accuracy of recommendations at the same time. Moreover, the continuous increase of the data received by the recommendation system is a dynamic process, and the application of DP in a continuous environment is still a problem. • The encryption-based solution can reduce the privacy risk caused by the leakage of user data. The current main method is homomorphic encryption (HE) [19] [20]. The use of a secure multi-party calculation protocol can ensure that the user input data is in an encrypted state, but still can perform accurate calculations, and ensure that the recommended accuracy is the same as in the non-encrypted state. Its disadvantage is that it needs to consume more computing time, storage space and communication cost, and it is only suitable for offline computing.

Federated Learning
FL following the "minimizing the amount of collection data" principle [21]. Compared with traditional machine learning methods that require training data to be concentrated in one machine or a data center, federated learning uses mobile devices (clients) dispersed in the hands of thousands of users to train machine learning models collaboratively, and all training data remains in their respective devices, thus protecting the privacy of users. In this way, avoid privacy risks and data security issues caused by centralized data collection and storage, but can still utilize all users data. The relevant background knowledge can be obtained from literature reviews [22][23][24] [10].
We here present the definition of federated learning as: Federated Learning is a learning framework in which multiple participants (mobile devices, organizations) holding different data collaborate to complete a specific machine learning task. A Host (a central server or an individual member) initiates a learning task, and under its coordination, each member trains the local data. Then, the initiator aggregates all encrypted training results safely as a global model, sends it back to the participants, and repeats the above process until the global model achieve the training objective. Finally, share the global optimized machine learning model with all participated members. A participant's raw data will not be exchanged or transferred throughout the process and ensure no participant can predict other members' original data. Set P as a performance measure (Accuracy, MSE, Precision, Recall, etc.). Denote P i as the performance of model M i , P Sum as the performance of model M Sum , and P F L as the performance of model M F L . A valid FL system will have P F L > P i , where i ∈ [1, N ], and its global objective is to minimize the |P Sum − P F L |, that is able to fully approximate the ideal model.
If there exist a non-negative real number δ makes the establishment of an inequality as following: It is said that the federated learning algorithm reaches δprecision loss [24]. FL allows a certain degree of deviation in the performance of the training model, but provides data security and privacy protection for all participants.  [36][37] to prevent internal and external privacy threats [38] and attacks [39]. These studies provide valid theoretical basis for introducing FL into the RS.
Ammad-ud-din et al. [40] introduced the first FL-based Recommender System to adopt the CF method (FCF) that utilized a stochastic gradient approach. Following FCF's work, Lin et al. [41] proposed a novel and generic federated recommendation model (FedRec) for rating prediction with explicit feedback. Muhammad et al. [42] proposed FedFast for making accurate distributed recommendations using DNN and FL. They adjust the Embedding's size to ensure its robustness on devices with insufficient computing power (mobile phone). Jalalirad et al. [43] introduced a simple and efficient extension of FL (SE-FRS) for recommender systems inspired by REPTILE meta-learning techniques [44]. Moreover, they confirmed the importance of utilizing embedding technology with an empirical study.
The FL show promising results in RS but still on the initial stage, existing studies mainly utilized feasible FL machine learning algorithms such as Stochastic Gradient Decent (SGD) [40] or DNN [45] to implement Collaborative Filtering based RS, and use HE and DP to optimized the data privacy protection in FL based RS [46]. These studies lack the overall consideration of the RS architecture. Treating all user data as private data is not practical and will encounter problems (unable to initialize model, data transmission overload, the insufficient computing power of user device) in a real application scenario. In response to these problems, we proposed the a novel privacy-preserved recommender system framework (PPRSF).

PPRSF
This section proposed our PPRSF (illustrated as Figure 2). We first present the preliminary work that initialize the member setting and data setting. Then, introduce the framework structure, the function, and the significance of each layer. Lastly, illustrate the FL based training method of essential models, the Global Recall Model, and the Local Ranking Model.

Preliminary
The federated learning system (FLS) developed by the Google team uses numerous users' smartphones as computing entities for algorithm training. This type of FLS constructed by a central server and massive terminal devices is called Centralized Cross-device FL Architecture [24] ,illustrated in Figure 3. The Cross-device FL system has a relatively large number of members (usually smartphone), and each member has small local raw data and relatively low computation power [8]. Therefore, this type of FL system relies on a mighty host to manage a stable communication network and cannot perform a complex machine learning task due to the limitation of member's computation power. Generally, in the form of a centralized network. Centralized Cross-device FL Architecture allows each member to train the model locally according to their conditions, aggregate the desensitization local model parameters on the central server, form a global model, and then send it back to each member. The member updates its local model until the global model has achieved the training object. In RS, the user's computer and smartphone as the client devices, and a central server (RS server) conducts data collection, model training, and online inference. The Centralized Cross-device FL architecture applies to the RS scenario.
In the traditional RS, the server will centrally collect the users' data and construct a RS model by training them together with the item data. Current FL based RS studies [40][41][42] [43] consider all the user data as private data, including explicit feedback (rating, comments). However, these data are public and are actively provided to the system by users. Some other user interaction behaviors, including page clicks, favorites, and likes, are also inevitably recorded by the system, as these behaviors are users' active requests on the application. Missing these data will make the RS unable to perform the model initialization (feature engineering, sample labeling), significantly increasing the client device's communication cost and computation load. The personal data that users do not want to disclose is private and need to be protected, such as personal information, personal health information, asset information, work information, and contact information. These data can indeed improve the RS's personalized recommendation capabilities, but these data are sensitive. Once malicious enemies obtain them will bring great distress and risks to the user.
Therefore, we divide user data into public user data (P ub_D) and private user data (P ri_D). The PPRSF central server can collect and further compute on the P ub_D. Including the user's typical interaction behavior in the application (Explicit Feedback, Implicit Feedback) and the data that the user actively discloses to the system, which is the essential private user data required by the application context (such as location-based recommender services require user's permission on real-time location data) based on a reliable agreement between the user and the application. P ri_D is personal data that the user consciously protects, and can only be locally preserved.
From the global perspective, assume N different clients U i have interaction records with the application, where i ∈ [1, N ]. On the client, each client device holds local raw data D i , contains public user data P ub_D i and privacy user data P ri_D i . As the features of local data are basically the same, and the user ID held by each client is different. Such data distribution in FL is called horizontal federated learning [24]. And the server holds size of M rows of global item data Global_I j as item inventory, where j ∈ [1, M ] , and collected user public data P ub_D i .

Framework Overview
The PPRSF including 3 essential models within 4-layers hierarchical structure. The system starts with a potentially huge inventory, massive item data is filtered from top to bottom through the layers of Recall, Ranking, and Re-Ranking, and the items that best meet user requirements and interests are presented to users through the client user interface.
Recall Layer -Server The Recall layer perform a rough sort on the huge inventory. Input pre-processed Global_I j and P ub_D i into the Recall model M Recall (U, I, Θ) , and generate relatively small candidate sets Recall_I ik for each user U i that recalled from a large inventory of items, each candidate set contain K rows of item feature data, where i ∈ [1, N ], j ∈ [1, M ], k ∈ [1, K], and K << M . And then send candidate set to each user's client device. The aim of Recall layer is to reduce the communication cost between server and clients, and the client's computational load during the local model training and inference phase.
Ranking Layer -Client [1-N] The Ranking layer exists in each client device. Each client device holds local private data P ri_D i and an independent local Ranking Model M Rank,i (U, I, θ i ). Each client U i receive Recall_I ik from server, and input into M Rank,i (U, I, θ i ) together with P ri_D i , the output is a fine-sorted candidate item list Rank_I ik accords with preference implied by the local private data, where i ∈ [1, N ], k ∈ [1, K]. The higher the ranking, the The local Ranking model training applied the FL paradigm, illustrated in section 3.3. Allows each client to perform the model training and inference on the local device and ensure that the private user data will not be acquired by the host server or any other clients. The Ranking layer aims to extend the performance of personalized recommendations while protecting user privacy.
Re-Rank Layer -Server The Re-Rank Layer on the server will receive the Ranked_I ik list request, then apply a supplementary strategy (based on application scenarios and system context), consider the diversity, freshness, popularity, and fairness of the final recommendation results before sending it to the client device. When the system encounters a new user or new items, the cold start strategy will perform this layer's task.
Service Layer -Client [1-N] The service layer presents the final recommendation results to each user through the client device's user interface. It is also at this layer that records user interaction behaviors, collects public user data to the server, and saves privacy-related user data on the client device. These records will participate in the next round of model training.

Model Training
In the PPRSF, the Global Recall Model on the server and the Local Ranking Model on each client are core modules that ensure the effectiveness, feasibility, and privacy of PPRSF. As illustrated in Figure 4, the server performs the training of the Global Recall model; once complete the training work, the output of the model (the recalled item data Recall_I i ) is sent to each client so that the client uses the independent recalled item data and the client's local private user data (P ri_D i ) to train the local ranking model. Subsequently, under the server's cooperation, each client's local models are aggregated on the server to form a global ranking model and then returned to each client for the local model update.
In Section 3.1, based on the definition of private user data and public user data, the server can perform data pre-processing (such as feature engineering, sample labeling, embedding) and jointly modeling on the P ub_D i and Global_I j , where i ∈ [1, N ], j ∈ [1, M ]. According to the application scenario and actual data condition, the Global Recall Model can be trained as a content-based model [47], or collaborative filtering based model [48], or neural network based model [49] [50]. In the absence of private user data, the Global Recall Model can guarantee to a certain extent that the recommendation results conform to the user's preference. After the training of Global Recall model is complete, each client is able to receive a filtered item inventory, which reduce the cost of communication. the host server start the Ranking model training task with global model initialization M Rank (U, I, Θ), then send initialized parameter Θ and Recall_I i to each client. In each client, perform local training work on Recall_I i and local private data P ri_D i , update the local model parameters θ i , and upload it to the server. As shown in Algorithm 1. return θ i to Server 10: end for 11: Local Ranking Model -M Rank,i (U, I, θ i ) As shown in Algorithm 2, after every client finished their local model training, the host server will receive the model update parameters θ i from each client, perform global aggregation to obtain global model parameter Θ, and return it to each client to optimize the local ranking model. After several iterations, each client obtained a global optimal model that able to perform local ranking work.

Conclusion
In this paper, we introduced the FL paradigm into the RS and proposed a novel Privacy-preserved RS framework (PPRSF) that enables the training and online inference under the premise of preserving user privacy, satisfying legal and regulatory requirements. This framework can be applied to the application that involving sensitive user information, such as financial service, medical service, and social network.