{'fraction_non_alphanumeric': 0.035329009549261804, 'fraction_numerical': 0.003748003780595118, 'mean_word_length': 5.17135961383749, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
PP-MARL: Efficient Privacy-Preserving MARL for Cooperative Intelligence in Communication

Artificial intelligence (AI) has been introduced in communication networks and services to improve efficiency via self-optimization. Cooperative intelligence (CI), also known as collective intelligence and collaborative intelligence, is expected to become an integral element in next-generation networks because it can aggregate the capabilities and intelligence of multiple devices. However, privacy issues may intimidate, obstruct, and hinder the deployment of CI in practice because collaboration heavily relies on data and information sharing. Additional practical constraints in communication (e.g., limited bandwidth) further limit the performance of CI. To overcome these challenges, we propose PP-MARL, an efficient privacy-preserving learning scheme based on multi-agent reinforcement learning (MARL). We apply and evaluate our scheme in two communication-related use cases: mobility management in drone-assisted communication and network control with edge intelligence. Simulation results reveal that the proposed scheme can achieve efficient and reliable collaboration with 1.1-6 times better privacy protection and lower overheads (e.g., 84-91% reduction in bandwidth) than state-of-the-art approaches.


I. INTRODUCTION
Artificial intelligence (AI) has witnessed rapid development in the past decade and has been applied for various contexts, including many mission-critical and real-time services for communication, such as Internet of Vehicles (IoV) [1]. This capability is largely owing to edge-and endpoint-AI (e.g., in smartphones and vehicles), which aims to reduce communication latency and preserve data privacy by moving AI closer to users. However, due to the limited capability in edges and endpoints (e.g., sensing, computing, and storage), it is still challenging for a single computing device to perform complicated tasks alone, such as autonomous driving. One of the promising approaches for addressing this challenge is cooperative intelligence (CI) [2], which integrates the capabilities of multiple devices to achieve joint goals and shared intentions. Specifically, CI is expected to facilitate next-generation networks by establishing collaboration among various communication-related intelligent equipment. In recent years, CI has been applied in various communication scenarios [3], such as adaptive routing, mobility management for communication devices, network control, and resource allocation.
Although CI provides several benefits in communication, various challenges emerge when deployed in the real world. The first challenge is to achieve coordination among intelligent devices. Notably, cooperative learning, such as cooperative multi-agent reinforcement learning (MARL) [4], has been exploited to address this challenge. Next, privacy concern is a prominent issue because existing schemes of cooperative learning are achieved by data and information sharing. For example, in aerial base station control, drones need to move cooperatively to achieve given goals (e.g., offer emergency communication) [5]. Due to privacy concerns, drones may be unwilling to share their data/information directly for cooperation (e.g., in-drone videos and movement trajectory). Another example is collaborative network management with edge controllers [1], in which edge information (e.g., customer locations, requirements, and feedback) is private. As a result, privacy issues become a major obstacle to the deployment of CI. Lastly, cooperative learning needs to be efficient with low overheads despite the practical constraints in communication (e.g., limited bandwidth and stringent latency requirements).
Several approaches exploit cooperative learning with (partial) privacy protection. However, they either lack efficiency by introducing high overheads [6] or provide inadequate privacy protection (e.g., vulnerable to inference attacks [7]). In this paper, to make CI deployable in practical communications, we propose PP-MARL, a novel cooperative learning scheme that 1) enables CI with privacy protection, in which the privacy-sensitive data (e.g., agents' observations and actions) is stored locally by introducing hierarchical critics and memories to MARL; 2) ensures the effectiveness of agents' collaboration through centralized learning by sharing less information than existing approaches; 3) exploits homomorphic encryption (HE) [8] and differential privacy (DP) [9] to further prevent the inference of private data from the shared information; 4) reduces overheads in communication and computation compared with existing schemes. Moreover, we apply and evaluate PP-MARL in two representative use cases: mobility management in droneassisted communication and network control with edge intelligence. Simulation results show that PP-MARL can achieve better collaboration than existing schemes with lower overheads and better privacy protection.

II. MARL FOR CI IN COMMUNICATIONS
In this section, we first introduce MARL for agents' coordination in CI. Next, we analyze challenges of applying MARL in communications through use cases studies. Last, we introduce existing privacy-preserving arXiv:2204.12064v1 [cs.MA] 26 Apr 2022 schemes for MARL and analyze their applicability to CI communications.

A. MARL for CI
CI is an integral function in multi-agent and distributed systems, where multiple intelligent agents seek collaborations to improve their welfare jointly. The concept of an intelligent agent, which originated from AI technology, is a computational unit that can complete a task or a part of a joint task autonomously, flexibly, and interactively. As illustrated in Fig. 1, many-fold entities can be regarded as intelligent agents in communication scenarios, including local, edge, and cloud agents.
One prominent way of coordinating agents to achieve CI in the long term is cooperative MARL [4], which is designed to find cooperative policies of agents by jointly evaluating their performance. The cooperative policies have advantages over independent strategies, especially when the agents can obtain only partial information (i.e., partial observations). MARL with partial observation can be formulated as a decentralized and partially observable Markov decision process (DEC-POMDP) [4]. DEC-POMDP is defined by five tuples, including states, observations, actions, transition function, and rewards. In DEC-POMDP, each agent observes the environment and gets observations which are partial information regarding the state of the environment. Next, agents make actions based on the observations to maximize their expected cumulative joint rewards. The state transition for producing the next state follows the transition function, which denotes probability distributions over states. In this process, the stored data of each agent is called experiences, including observations, actions, rewards, and successive observations.
In MARL, three cooperation modes have been proposed to incentivize agents' cooperation by sharing information, negotiating, and learning cooperatively, depicted in Fig. 2. Firstly, the centralized mode relies on a central node to coordinate the training and even the execution of agents, such as MADDPG [10]. The central node should belong to a trusted authority to make and enforce decisions on behalf of a swarm of agents; otherwise, agents will not trust the central node, such that the agent cooperation will be obstructed. Therefore, the centralized mode suffers from a single point of failure and privacy issues (i.e., sharing data with a central node). Secondly, in the decentralized mode, agents are independent learners and workers with no explicit information exchanged with each other, such as DDPG [11].Strictly speaking, this mode is non-collaborative, and its non-stationary characteristics in partial observation environments may lead to non-convergence of the learning [10]. Lastly, the networked mode is a decentralized model with networked agents, in which agents are connected via a communication network so that their local information can be spread across the network, such as FedQ [6]. However, this mode scales poorly with an increasing number of agents because its communication overheads increase dramatically. In addition, information sharing among agents poses privacy issues.

B. Use Case Studies
To understand the challenges of applying MARLassisted CI in communication, we study two typical multi-agent use cases as illustrated in Fig. 1.
1) Mobility management in drone-assisted communication: In this use case, drones, working as aerial communication infrastructures, provide network services to targets (e.g., vehicles and users) [5]. Each drone has limited coverage and cannot serve all targets anytime and anywhere. Therefore, multiple drones need to cooperate to cover targets while avoiding collisions. Drones, also working as agents, can observe positions of targets and other drones in range and then decide movements to maximize the joint rewards (e.g., the number of covered targets) cooperatively. Given that each drone has a partial observation of the environment, it would be difficult to make a wise movement without cooperation. However, data sharing to support cooperation brings a privacy risk (e.g., revealing the location of vehicles) and introduces additional communication overheads (e.g., in terms of bandwidth and energy consumption).
2) Network control with edge intelligence: This use case is CI for network control in hierarchical softwaredefined IoV (HSD-IoV) [1]. HSD-IoV consists of edge controllers located in base stations (BSs) and a core controller in the cloud. The edge controllers are close to vehicles and offer low-latency network control, while the core controller can coordinate the learning of edge controllers in the centralized mode. The problem is how to assign edge controllers to vehicles dynamically and cooperatively to minimize the latency of responses. The edge controllers, working as agents, observe the demands and locations of vehicles in their observation range. Then, they make decisions locally regarding the edge controller assignment policy (i.e., to find a good matching between edge controllers and vehicles) based on their observations, which can be viewed as actions. Due to agents' limited observations, they need to assign edge controllers to vehicles cooperatively to minimize the overall latency. The detailed formulations can be found in our previous work [1]. Similarly, data/information sharing for cooperation also brings risks in privacy leakages, such as vehicles' locations and demands. In addition to bandwidth overheads, agent cooperation during action-making (i.e. execution) results in delays which cannot be ignored, as network control is a delay-sensitive task.
Challenges. According to these use case studies, CI has the potential to improve communications (e.g., in mobility management and network control). However, it also brings new challenges in privacy protection and overheads. Given the practical constraints (e.g., privacy concerns and limited bandwidth), the CI used for communication problems needs to be efficient privacy protected, which includes 1) privacy protection over agents' experiences, 2) low overheads for cooperative learning and execution (e.g., bandwidth, energy, latency and computation), and 3) ensuring the gains of collaboration.

C. Existing Privacy-Preserving Schemes for MARL
According to the use case analysis in §II-B, we find that agent cooperation encounters a practical challenge of privacy protection: most existing MARL schemes share experience directly for learning (e.g., MADDPG), leading to potential leaks of agents' data. Recently, some state-of-the-art MARL schemes have been proposed to offer (partial) privacy protection.
1) Independent learner: This straightforward scheme has been used in proposals like DDPG, where all agents execute and train independently. Strictly speaking, this scheme does not offer any data sharing and is noncollaborative. The lack of collaboration poses a convergence problem in agents' learning because the par-tially observed environment without data sharing is nonstationary [10].
2) Federated learning (FL): FL [12] can protect the privacy of the local data, which allows multi-agents to train with local data respectively and build a shared model by sharing the model parameters or gradients. Solutions integrating FL into MARL have also been proposed, such as FedQ. FedQ federatively builds global models using networked cooperation modes. However, FedQ incurs higher communication overheads and delays in agents' execution because its training and execution rely on information sent from neighbour agents. Therefore, the performance of FedQ is not acceptable in timesensitive scenarios.
3) Data encryption: This approach converts raw data to encrypted data, such as through HE [13] and DP [14]. HE has been introduced for data encryption in single-agent reinforcement learning [13]. However, the efficiency issue is the most challenging in deploying HE for MARL because the HE scheme incurs significant computational overheads for supporting encryption on the shared data. The overhead is especially high for traditional MARL due to their high volume of shared data (e.g., all experiences in MADDPG and observations in FACMAC [7]). Next, DP [14] is another technique that protects data by injecting noises. DP has lower overheads than HE, but DP brings a trade-off between cooperation performance and privacy protection.
III. PRIVACY-PRESERVING MARL FOR COMMUNICATION As mentioned above, the existing schemes cannot address all the challenges mentioned in §II-B simultaneously as they either lack collaboration, have inadequate privacy protection or introduce high overheads. In this section, we propose PP-MARL to offer efficient privacy protection for CI given the practical challenges in communications.

A. Overview of PP-MARL
In a nutshell, PP-MARL is designed in a way which allows decentralized execution with localized actors and centralized training with hierarchical critics and memories, as presented in Fig. 3. Decentralized execution offers low latency, which meets the requirements of latency-sensitive applications in communication; centralized training with a central node for coordination training guarantees convergence; hierarchical critics and memories support efficient privacy protection.
The execution is an interactive process between agents and environment, during which agents can get and store data (i.e. experience) for learning. The execution is designed to be decentralized in PP-MARL, enabling low-latency actions by decoupling dependencies from other agents. Firstly, agents independently (and possibly partially) observe the environment and obtain local observations (e.g., the position of vehicles within the observation range). Next, each actor in the agents makes an action decision given its local observation. All the actions are aggregated into joint actions, and then the joint actions act on the environment (e.g., drones move to target places and offer services to vehicles). Then, the environment gives feedback (i.e., rewards) to the agents given their actions and environment state, and the environment transfers to new states. Lastly, agents' experiences are hierarchically stored for privacy protection. The global memory stores public data (i.e., memory identifier), and each agent stores private experiences into its local memory, including local observations, individual actions, rewards, and successive observations.
The training in PP-MARL is designed to be centralized by introducing hierarchical critics to estimate the value of joint action-values. A hierarchical critic consists of local critics of the involved agents and a global critic for local value integration. The local critic in each agent evaluates local performance on the local observations and actions, and then it gives local critic values (abbreviated as q values) to the global critic.Strictly speaking, a local critic is a utility function, instead of a value function, because it cannot estimate expected rewards by itself, and therefore q values cannot be used for updating the actors directly. The global critic is located in the central node and designed as a deep neural network. It combines q values of all involved agents to evaluate the joint actions based on the joint observations and gives global critic value (abbreviated as Q values). As agents may have different reward functions, multiple global critics can be introduced for different agents or teams in the central node.
The hierarchical critic differs from a decentralized critic (e.g., DDPG) and a centralized critic (e.g., MAD-DPG). A decentralized critic estimates the local actionvalues based only on local observations and actions for each agent. On the other hand, centralized critics can use the information of all agents to estimate the joint actionvalues based on the joint observations (e.g., MADDPG) or environment states (e.g., FACMAC). However, the centralized critics introduce high communication overheads and privacy issues. For example, the critics in MADDPG are trained with experiences that include observations, actions, and rewards of all agents. To avoid high overheads and privacy leakage in training, PP-MARL introduces hierarchical critics so that only q values are shared to estimate the joint action-values. The centralized training process for actors and hierarchical critics in PP-MARL is described in detail below.

1) Hierarchical critic updates:
A hierarchical critic can be viewed as a tree-like neural network whose parameters are updated to minimize the loss function over Q values. Firstly, the central node provides sampled identifiers to all involved agents for memory extraction.Secondly, local critics give corresponding q values to the global critic. Next, estimated rewards of each agent are calculated by the central node, which are the distance between Q values and discounted target critic values (i.e., the estimated further accumulated rewards) of the next step. The estimated rewards are then sent back to agents. After receiving the estimated rewards, each agent calculates the loss defined as the mean squared error between the estimated rewards and the locally stored rewards (i.e., the ground truth) and sends the loss back to the central node for updating the hierarchical critics. Finally, the hierarchical critic is updated with backward propagation.
2) Actor-network updates: The parameters of the actor-networks are updated using the sampled policy gradient [10]. Policy gradient methods optimize parameterized actors with respect to the Q values by gradient descent. The Q value is obtained by merging q values with global critic networks. The local critic in the agent to be updated is fed with actions obtained by its actornetwork and local observations from its local memory. The other local critics are fed with local observations and actions from their local memories.

B. Privacy Protection in PP-MARL
Through PP-MARL architectural design, we are able to reduce the amount of information for sharing and transform agents' experiences to q values for privacy protection. Furthermore, to avoid inferring private data through the q values, we introduce HE and DP to further improve privacy protection, as depicted in Fig. 3. PP-MARL has an HE-friendly architecture because the amount of shared data (i.e., q values) to be encrypted is much less than existing schemes. In contrast, if HE is used in MADDPG, it needs to encrypt over observations, actions, and rewards, and FACMAC needs to encrypt over q values and states.
Deploying HE to neural networks meets challenges because HE schemes support only homomorphic arithmetic operations such as homomorphic addition and multiplication. However, most of the popular and standard activation functions in neural networks are nonarithmetic, such as rectified linear units (ReLU) and sigmoid [8].To address this issue, we introduce interactive HE, which executes non-arithmetic operations (i.e., activation functions) locally by agents or a trusted device with the secret key (see green dash lines in Fig. 3). Interactive HE operations include: encrypting q values, decrypting the outputs of middle layers for activation, encrypting the data after activation, and decrypting predicted rewards to calculate losses. Clearly, this process brings additional overheads (e.g., bandwidth and energy) in data encryption, decryption and transmission. To mitigate this, one approach is to replace the non-arithmetic activation function by the low order polynomials or linear activation function [15]. With such a mitigation approach, the overheads in PP-MARL will be much lower without the need to decrypt, encrypt, and transfer the outputs from hidden layers (i.e., processes with green dash lines). However, there is a trade-off between cooperative learning performance and overheads. Besides HE, DP is another potential technique for privacy protection. By adding noise to the q values, it can improve privacy protection for PP-MARL as well. However, DP methods often have to tradeoff the degree of privacy protection for user data against the model's accuracy [9], as also confirmed by our evaluation results (see Table I). The detailed comparison will be discussed in §IV In brief, PP-MARL offers better privacy protection than existing approaches because more private data (i.e., local observations, actions, and rewards) are stored and used locally. Furthermore, HE and DP are introduced in PP-MARL to improve privacy protection against data inference attacks further. Next, PP-MARL is more efficient as the exchanged messages between agents and the central node are less than the existing approaches. The specific comparisons are described in §IV.

IV. EVALUATION AND ANALYSIS
This section shows the performance comparison of PP-MARL with existing approaches in the two use cases discussed in §II-B in terms of the learning performance, privacy protection degree, and overheads in communication and computation.

A. Drone-assisted Communication
In this experiment, three drones are employed to cover three target places to provide communication services. The drones serve in rectangular space of 1×1 km 2 . The positions of drones and targets are initialized randomly. Each target place is a circular space with a radius of 40 m, and the coverage radius of drones is 50 m. Each episode is divided into 10 identical time-intervals (i.e., steps). Drones aim to cover all the targets as soon as possible cooperatively.

Low
Medium High   4 shows the learning performance comparison with four representative algorithms. The y-axis is the mean of the total number of target places covered by drones in each episode, and the x-axis is the ability of agents' observations, which ranges from low to high. We can observe that cooperative learning schemes (i.e., PP-MARL, FACMAC, FedQ, and MADDPG) generally achieve better performance than decentralized learning schemes without cooperation (i.e., DDPG). The former operates in either the centralized mode or the networked mode to share information (i.e., q values in PP-MARL and FedQ, global states in FACMAC, and agents' experiences in MADDPG) in training, while the latter lacks cooperation in learning, resulting in a worsened training performance. While individual intelligence alone may be limited, collaboration in learning can bring together agents' intelligence, leading to better performance for the application. Generally, PP-MARL achieves better performance than the others, except the one with a high observation range where FACMAC has a limited advantage over PP-MARL. Furthermore, by comparing the different observation ranges, we can find that a broader range of observation does not necessarily lead to better performance for all MARL algorithms. This phenomenon is shown in PP-MARL and DDPG, for which the medium observation is the best. A wider range of observations can provide more information for the agents in acting while bringing difficulty in training owing to the larger input dimension. I: Index comparison: cooperation mode, private data, public data, privacy protection, learning performance (named as gains), and overheads including communication (the size of messages to be delivered, in MB), energy and computation relative to DDPG. w stands for weights of neural networks. Data with a tilde is encrypted. h is the coefficient for HE operation cost relative to the multiplication operation.

B. Network Control with Edge Intelligence
For HSD-IoV, we use the dataset of vehicle mobility traces from SNDlib [1]. This dataset offers real-time position data reported by buses from Rio de Janeiro (7 days in 24-hour format). The region is 10×10 km 2 map of Rio de Janeiro, Brazil. There are 4 BSs equipped with edge controllers and they all have a coverage radius of 4 km. The observations of the edge controllers varies based on their locations, and their service coverage may overlap. The problem is how to assign edge controllers adaptive to loads of vehicles in the overlap zone and minimize the response latency. We show simulation results of PP-MARL compared with FACMAC, MADDPG, DDPG, and two non-intelligent approaches (i.e., DB and RC [1]). DB is a distance-based greedy policy. RC is centralized intelligence using a cloud controller.  it can be demonstrated that edge intelligence has less control delay than centralized intelligence (i.e. RC). DB has a higher delay than the MARL algorithms, as it only considers distance and ignores the load of edge controllers. DDPG has limitations in delay reduction compared with cooperative MARL algorithms due to lacking cooperation. In summary, CI can provide flexible network management, and PP-MARL can further reduce control delays by efficient cooperation.

C. Privacy Protection and Overheads
In addition to evaluating learning performance, we compare the privacy protection and overheads of PP-MARL with baselines for IV-A in Table I.
Firstly, the privacy protection degree is defined as the root mean square error (RMSE) between the inferred data and actual ones. A higher RMSE implies that it is more difficult to infer private data from public data and thus able to provide better privacy protection. The results reveal that DDPG provides the best protection because it is non-cooperative without information sharing, and the inferred data are predicted from zero vectors as inputs. However, as our experimental results in §IV-A and §IV-B show, non-cooperative MARL (i.e., DDPG) can be less effective with lower gains. PP-MARL provides better privacy protection than the other cooperative methods because it only shares q values. Although valuedecomposition methods (e.g., FACMAC) can protect privacy for actions, private data may still be inferred much easier from their public data. FedQ provides marginally less privacy protection than PP-MARL owing to reward sharing.
Secondly, we compare PP-MARL with baselines in terms of bandwidth, energy, computation, and delay overheads. The results show that PP-MARL has obviously low overheads than the other cooperative MARL. The results are consistent with our previous analysis in §II-C: DDPG has no additional overheads as it is a noncooperative; MADDPG introduces higher bandwidth due to more data sharing between agents and the central node; FedQ has a disadvantage in supporting low-latency applications because of higher delays and energy cost in execution, and clearly it introduces high communication and computation overheads; FACMAC has lower overheads than FedQ and MADDPG, but is still significantly higher than PP-MARL due to shared states.
Thirdly, we show the performance and overheads of HE and DP deployed in PP-MARL, as shown in Table I. The results show that introducing interactive HE (i.e., HE 1 ) into MARL brings obviously high overheads during training. For example, MADDPG with HE 1 introduces around 10 times higher bandwidth cost and 3 times more energy consumption during training. In PP-MARL, the increment by introducing interactive HE cannot be ignored, even though PP-MARL is designed to be HE-friendly by reducing native communication data volume (only sharing q values) and decomposing the critic neural network to be hierarchical. Furthermore, we considered linear activation function as a tradeoff scheme for PP-MARL (i.e., HE 2 ). The PP-MARL-HE 2 has obviously lower overhead than PP-MAL-HE 1 , but at the cost of the lost gain in cooperation. PP-MARL-DP shows the decrease in gains, privacy protection, and computational overheads, which adds Laplaciandistributed noise [9] on the q values. Hence, we prefer HE to DP to improve PP-MARL privacy protection, given that computing and energy are not strictly limited at the central node. Moreover, we prefer interactive HE only when the agents (e.g., edge and end devices) is powerful.
From the above comparison, we can draw the following conclusions. PP-MARL provides the best privacy protection with the lowest overheads among all the cooperative MARL schemes. Besides, we introduce PP-MARL-HE and PP-MARL-DP, variants of PP-MARL, which further improve privacy protection at the cost of higher overheads in communication and computation than PP-MARL. In addition, PP-MARL proves to be an HE-friendly architecture, introducing less overhead than the other MARL schemes with HE.
V. CONCLUSION AND DISCUSSION In this paper, we exploited CI in facilitating nextgeneration networks. We showed that the collaboration among agents was achieved by data and information sharing via communication. However, given the privacy concern and practical constraints in communication, the deployment of CI was obstructed. To address this issue, we proposed a privacy-preserving MARL scheme, called PP-MARL, which leverages an HE-friendly architecture. We evaluated PP-MARL's performance in two representative use cases: mobility management in drone-assisted communication and network control with cooperative edge intelligence. Simulation results revealed that cooperative learning promotes collaboration among agents, and PP-MARL can provide better privacy protection with lower overhead than state-of-the-art approaches.
Privacy protection for CI in communications remains challenging in environments with large-scale agents or a variable number of agents involved in the collaboration. In future research, we will extend PP-MARL to support communication problems with dynamically varying numbers of large-scale agents, such as network control with mobile edge controllers.