{'fraction_non_alphanumeric': 0.03241810163796724, 'fraction_numerical': 0.010499790004199917, 'mean_word_length': 4.692066337965038, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reconfigurable Intelligent Surface Assisted Device-to-Device Communications

Reconfigurable intelligent surface (RIS) technology is a promising method to enhance wireless communications services and to realize the smart radio environment. In this paper, we investigate the application of RIS in D2D communications, and maximize the sum of the transmission rate of the D2D underlaying networks in a new perspective. Instead of solving similarly formulated resource allocation problems for D2D communications, this paper treats the wireless environment as a variable by adjusting the position and phase shift of the RIS. To solve this non-convex problem, we propose a novel double deep Q-network (DDQN) based structure which is able to achieve the near-optimal performance with lower complexity and enhanced robustness. Simulation results illustrate that the proposed DDQN based structure can achieve a higher uplink rate compared to the benchmarks, meanwhile meeting the quality of service (QoS) requirements at the base station (BS) and D2D receivers.


I. INTRODUCTION
As one of the key technologies of the fifth-generation (5G) and beyond communication systems, underlaying device-todevice (D2D) communications permit devices to communicate with proximity devices over the licensed spectrum for cellular networks, thus enhancing the communication system performance by reducing the latency, improving energy efficiency (EE), and spectrum efficiency (SE) [2]. D2D communications have been applied in various applications, including the 3rd Generation Partnership Project (3GPP) proximity services [3], Internet of Things (IoT), vehicle-to-everything (V2X) communications, and wearable communications [4]. According to the expectation of Cisco, the share of D2D links will increase 20 percent from 2018 to 2023 [5]. These various applications pushes the development and implementation of D2D communications.
There is a rich body of literature that focuses on the resource allocation for D2D communications [6]- [8]. Recently, a novel approach referred to as the smart radio environment presents a new perspective to enhance the D2D communications. Particularly, the wireless environment can be controllable and programmable. In this way, we can optimize the communication environment and resource allocation for D2D devices simultaneously, thus permitting us to control or eliminate the interference significantly.
One key technique to realize the smart radio environment is reconfigurable intelligent surfaces (RISs) [9], which have attracted extensive attention in wireless communications. Equipped with an array of low-cost passive reflecting elements, the phase shift and reflection amplitude of each RIS element can be adjusted by a controller, enabling it the ability to modify wireless communication environment proactively. Compared with the conventional relays, the advantages of RIS include lowered energy consumption and enhanced system capacity [10]. Although the control signal can be analog using varactors to achieve continuous phase shift [11], the long response time and low phase accuracy of varactors make it impractical for wireless communications. Theoretical analyses have been provided for multi-bit controlled elements to strike a tradeoff between the system performance and the complexity [12], [13]. The performance improvement has been further verified by a RIS-based wireless communication prototype [14], which motivates us to apply it in D2D communication systems.
On the other hand, the large number of RIS elements requires optimization approaches with lower complexity. Although typical optimization tools such as exhaustive search can generate the optimal solution, the high computational cost makes it unrealistic for the real-time optimization. Fortunately, machine learning (ML) methods, especially deep learning (DL) approaches, have become promising tools to address nonlinear non-convex problems and high-computation issues, which are mathematically intractable. Particularly, deep Qnetwork (DQN) has shown its power in solving sophisticated decision-making problems under uncertain and dynamic environments, e.g., human-level game playing [15], [16] and AlphaGo [17]. Inspired by the remarkable performance of DQN in various areas, there have been some works exploring its application in wireless communications. DQN provides a principled and robust method to tackle the dynamic environment by making decisions for discrete optimization problems, which bring it the ability to optimize the resource allocation for D2D communications in varying channel state environment. Moreover, as a new technique of DQN, the proposed DDQN provides a more reasonable way to evaluate and execute the action, which avoids the overestimation challenge of legacy DQN algorithms and is more robust to time-varying environment [18].
A. Related work 1) Resource allocation in D2D: The existing works on D2D communications mainly focus on transmit power and arXiv:2107.02155v1 [eess.SP] 5 Jul 2021 channel assignment optimization [6]- [8]. In [7], EE is maximized by optimizing the transmit power while satisfying the QoS requirement for D2D and cellular users. To overcome the challenges caused by dynamic D2D channels, a co-design of robust spectrum allocation and power optimization has been proposed [8]. While there exists plenty of literature applying optimization tools to solve resource allocation problems for D2D communications, most of them requires intensive computation at the BS to run the optimization algorithms [19].
As discussed earlier, DL based approaches enable wireless communication users to treat the dynamic environment and make their robust decisions with lower computational complexity. DL has been applied to physical layer processing [20] and resource allocation [19]. Moreover, relying on the local users information and observations, multi-agent reinforcement learning (MARL) based decentralized optimization approaches have been widely applied in wireless communications [21]- [24]. Leveraging MARL, D2D users can make their own decisions on transmit power and spectrum sharing policy, which also offloads the computational complexity from the BS to users. In [21], each D2D pair is a learning agent and able to explore the unknown environment. Each D2D user chooses its transmit power level and sub-channel to minimizing long-term system cost. However, the unknown policy and information of other users causes a non-stationary environment. To overcome it, MARL algorithms with improved state observation have been proposed in [22], [23] to perform spectrum sharing. Particularly, the D2D links could be deployed as MARL agents, learning to access the channel of cellular users by collectively interacting with the communication environment and receiving the rewards. Such decentralized optimization approach has been verified to achieve the near-optimal performance [24]. Although the above valuable works improve the performance of D2D communications significantly, they mainly focus on the transmit power allocation and channel assignment allocation under fixed communication environment. With the help of RIS, we are able to actively control the communication environment and optimize the resource allocation from a brand new perspective.
2) RIS enhanced wireless communications: Recently, RIS has been explored in a wide range of scenarios, e.g., RISenhanced cellular networks beyond 5G, RIS-assisted indoor communications, and IoT applications [25]. Particularly, RIS has been successfully applied in D2D networks in [26]- [28]. Many approaches have been developed to optimize RIS for achieving higher throughput or EE. To solve the non-convex maximizing problems, they tend to find sub-optimal solutions by using the block coordinate descent [26] and Riemannian pursuit method [27]. To achieve a performance-complexity tradeoff, the projected sub-gradient method is adopted for the phase shift [28]. However, to enhance the overall system performance, the optimization of RIS becomes a challenge due to the huge number of reflecting elements to optimize [29]. The time-varying D2D channel also brings high complexity to optimization algorithms.
A well-trained ML model is an effective approach to lower computational cost. Although the ML model requires more computations at the training stage, it could be trained offline and is robust to dynamic environment. As a novel branch of ML, DL has been applied for channel estimation and phase shift optimization in RIS-aided communications [30]. Motivated by the applications of DL in solving sophisticated optimization problems, the authors in [31] have applied the DL method for estimating the channels and configuring of RIS. Moreover, DQN has shown its potential in various communication scenarios, e.g., massive multiple input multiple output (MIMO) systems [32], RIS-aided unmanned aerial vehicle (UAV) [33], and non-orthogonal multiple access (NOMA) [34].

B. Motivation and Contribution
We consider a reconfigurable intelligent surface enhanced D2D communications system to actively optimize the performance of communication. The challenges occur in several aspects. The fast channel variations of D2D communications makes the conventional resource allocation approaches based on stable channel state information (CSI) not applicable anymore [8]. Most of the current works for the RIS enhanced D2D communication separate the RIS optimization and resource allocation into sub-problems, then leveraging alternating optimization to solve the problem [26]- [28], [35]. The large number of RIS elements and alternating optimization means high computational complexity. Additionally, all of these works only consider the phase shift, i.e., passive beamforming design, while neglecting deployment position of the RIS. To overcome the challenges, a low complexity approach is required to achieve real-time optimization [30].
In this paper, we apply a DDQN based structure to optimize the transmit power of D2D users, the channel assignment for D2D pairs, the RIS position, and the phase shift. The major contributions of this paper are summarized as follows.
1) The position and phase shift of RIS are jointly optimized at the BS, while the decision-making of channel assignment for D2D links is performed decentrally based on their observation, thus offloading the computational pressure at the BS and enhancing data privacy. 2) A novel reinforcement learning structure is proposed to execute the resource allocation, the phase shift, and RIS position deployment task with lower computational complexity and communication cost. To enhance the robustness and effectiveness of the proposed algorithm, a DDQN algorithm is applied to overcome the overestimation problem. 3) Based on numerical results, the proposed decentralized DDQN architectures can achieve near-optimal performance with low complexity and high robustness.
The rest of this paper is organized as follows. The system model of RIS enhanced D2D communication system is presented in Section II. Then, the decentralized resource allocation structure and the DDQN based centralized RIS optimization are introduced in Section III and Section IV, respectively. Simulation results are presented in Section V. Finally, conclusions are drawn in Section VI.

II. SYSTEM MODEL
In this Section, a RIS enhanced D2D network model is described and an uplink rate maximization problem is formulated.

A. System settings
We consider the uplink transmission in a cellular network, which includes K cellular users communicate with the BS in the cellular mode, and I D2D pairs communicating with each other by sharing the resource blocks (RB) with cellular users. Assuming that the i-th D2D transmitter, D t i , communicating with the corresponding receiver, D r i , by reusing the RB assigned to the k-th cellular user, U k , then D t i becomes the source of interference for U k . To enhance the transmission performance of the network, a RIS composed of N passive elements is deployed.
As shown in Fig. 1, we consider that all channels consist of large-scale fading and small-scale fading. The direct links, i.e., the links without the aid of the RIS, are modeled by the Rayleigh fading. The channel gain g D i [k] between the D2D pair (D t i and D r i ) over the k sub-band, which is preoccupied by the k-th cellular user U k can be denoted as where f N LoS [k] is a frequency dependent variable that represents small-scale Rayleigh fading power component distribution for k-th sub-band. Meanwhile, the large-scale path loss is assumed to be frequency independent and is modeled as is the distance between D t i and D r i , and α represents the path loss exponent. Similarly as (1), the channel gains for the direct link between U k and D r i , the link between U k and the BS, the link between D t i and the BS over k-th sub-band are denoted by g k,i [k], g U k [k], and g BS i [k]. Due to the existence of line-of-sight (LoS) links, we model the RIS aided links as Rician fading channels. The channel gain h[n, k] of the RIS aided link between users to the n-th RIS elements is denoted as where v is Rician factor and f LoS [k] refers to Rician fading power component distribution for the k-th sub-band. We define the channel gain h Dt i [k] for the link between the i-th D2D transmitter D t i and the RIS as where h(·) T represent transpose matrix of h(·), h Dt i [n, k] could be defined as (2), representing the channel gain between D t i and the n-th RIS element. Similarly, the channel gains for the link between the RIS and D r i , the link between U k and RIS, the link between RIS and BS are defined (3), respectively. The phase shift and amplitude attenuation A for all the RIS elements can be expressed as Overall, the channel gain over the k-th sub-band for the i-th Similarly, the overall channel gain between U k and D r i , the channel gain between l-th D2D transmitter D t l and D r i , the gain between U k and the BS, the gain between D t i and the BS can be represented as , respectively. The signal y i [k] received by D r i over the k-th sub-band is denoted as i and U k , p D i and p U k denote the transmit power of the D t i and U k , and u D i and u U k represent the unit variance entries with zero mean, and z ∼ N (0, σ 2 ) denotes the AWGN noise signal with mean 0 variance σ 2 . Then, the signal-tointerference-plus-noise ratio (SINR) at D r i and the BS for U k over the k-th sub-band can be denoted as and respectively, where ρ k,i is the resource reuse coefficient of U k and i-th D2D pair, and ρ k,i = 1 when i-th D2D pair reuses the channel assigned to U k . Otherwise, ρ k,i = 0. Moreover, the interference to D r i is given by Then, the ergodic capacity for i-th D2D pair and for the k-th cellular user U k can be denoted by and respectively, where E[·] represents the statistical expectation of [·], representing the expectation of the rate over the small scale fading distribution, B k is the bandwidth of k-th sub-band. The channel capacity of underlaying D2D networks could be expressed by

B. Problem formulation
We aims to maximize the long-term sum rate in (6)  maximize where γ D min and γ U min are the minimum SINR requirements at the D2D receiver and the BS, respectively. Coordinate s RIS restricts a two dimensional (2D) space for the installation of the RIS. Constraint (12d) assumes that each D2D pair only occupies one RB. Due to hardware limitations, RIS elements can only provide discrete phase shifts. This constraint and (12d) make (P1) non-convex. To solve the non-convex problem, we have to utilize exhaustive search, which is impractical when the number of D2D pairs, cellular users, and the number of RIS elements become large. Generally, classical optimization tools can be leveraged to acquire suboptimal solutions [26]- [28]. Alternatively, instead of solving challenging non-convex problem by mathematical tools, we leverage an deep RL based algorithm, which is more applicable to solve problems with high dimension inputs as well as large state and action space which will be detailed in Section III.

MULTI-AGENT REINFORCEMENT LEARNING
In this section, we study the transmit power optimization and channel assignment optimization for each D2D pair. The optimization objective is to jointly optimize the resource allocation for D2D pairs, plus the position and phase shift for the RIS. Instead of optimizing the configuration of RIS and the resource sharing centrally at the BS, we propose the decentralized resource sharing scheme and centralized RIS optimization approach. By decoupling the joint optimization into sub-problems, we not only reduce the computational pressure of the BS significantly, but also enable the D2D users to determine their resource sharing policies by local information.
As shown in Fig. 1, D2D pairs reuse the RB occupied by cellular networks. Given an arbitrary RIS implementation, i.e., the position and phase shift of the RIS, the resource allocation optimization problem can be simplified into

A. System description
Generally, the resource allocation optimization problem can be modeled as a linear sum assignment programming (LSAP) problem and can be solved by Hungarian algorithm [36] with computational complexity O(n 3 ). The complexity is much higher if we take the transmit power of D2D users into account. The high complexity of the Hungarian algorithm makes real-time optimization is impractical in the proposed D2D communications scenario. Additionally, the algorithm needs to be robust for fast channel variations and unstable CSI for different RIS implementations. Leveraging DDQN, we can train the agents under different CSI conditions so that it can be adapted to the various communications system.
Then the channel assignment and transmit power optimization problem can be modeled as a MARL problem. It is noted that the DDQNs at the D2D users is should be trained offline. Actually, updating the resource allocation policy too quickly can cause challenges on convergence performance when we train the DDQN for RIS optimization. This is because even if the RIS controller takes the exactly same action, the rewards would be various for different resource allocation policies, making the algorithm hard to converge. The unstable reward requires a robust resource allocation algorithm so that it can works under different RIS implementations.

B. Concept of Reinforcement learning and deep Q-network
RL is a branch of ML paradigm that allows agents to learn the optimal policy by the trial-and-error interaction with the environment to maximize the desired reward. Mathematically, the RL can be modeled as an markov decision process (MDP), including environment state S, actions A, and the reward R which can be determined for each state-action pair. During each time slot t, each agent observes the state s t ∈ S and then take an action a t ∈ A according to a certain policy π. Then the agent receives the corresponding reward r t and turn to the next state s t+1 , which is determined by action a t but independent of the past states. Formally, this process can be denoted by a transition tuple e t = (s t , a t , r t , s t+1 ). The interaction process is shown in Fig. 2.
During each time slot t, the objective of RL is to maximize the cumulative desired return from time t to the future, which can be expressed by where γ ∈ (0, 1) represents the discount factor which represents the impact of the future reward. The expectation reward for a state-action pair (s, a), the action-value function, is defined as where policy π is defined as a mapping from state S to the probability of choosing each action in A.
The objective of RL is to find a optimal policy π * = arg max π q π (s, a). The optimal action-value function obeys an important identity known as the Bellman equation. The optimal policy is to select the action that maximizes the expected Q-value [15]: Authors in [37] have shown that, q(s t , a t ) → q * (s t , a t ) as t → ∞. However, it is impractical since the iteration is discrete. Instead, the NNs are applied to be function approximator to estimate the action-value function, i.e., q(s t , a t ; W ) ≈ q * (s t , a t ), which is the basic idea of the DQN. When the state and action space become large, this method does not need to maintain the large Q-table as conventional RL approaches do, thereby expanding the applications of RL in wireless communications greatly. The training data set, also named replay memory D = [e 1 , . . . , e t , . . . ] for NN is stored according to agent's experience at each iteration t, where the experience e t = (s t , a t , r t , s t+1 ) is called transition, including the state, action and reward information. The training minibatch (s j , a j , r j , s j+1 ) is sampled from the training data set. During the training process, parameters are updated to the Q estimation network at each step to generate the estimated Q-value. Q target network is updated after every g steps according to the parameters in the Q estimation network. The training process for DDQN is to minimize the error function which represents the estimated Q-value and the realistic Q-value. For the DDQN in this work, the error function can be expressed by: where q target = r j + γ max a q(s j+1 , a ; W − ) is the target Q-value for minibatch j, which is the output of Q target network, W and W − denotes the weights of the evaluation network and the target network, respectively. The weights are optimized by the gradient descent method [15].

C. Double DQN algorithm
The DQN algorithm can achieve a near-optimal performance in some scenarios, while sometimes it causes the overestimate problem. The target Q-value is approximately generated by the target network by maximizing the actionvalue function, while this target value is even higher than the true optimal action-value. The overestimate problem is severest when the number of actions becomes large, affecting the convergence and performance of the learned strategies. The idea of DDQN is decomposing action selection and action evaluation [18] to reduce overestimations. Unlike DQN that uses the evaluate network to estimate the action-value function and select action at the same time, DDQN uses the target network when evaluating the action-value function. In other words, the DDQN uses the evaluate network to select the action, while using the target network to fairly evaluate this action. The updated target Q-value function in DDQN is defined as q target = r j +γq(s j+1 , arg max a ∈A q(s j+1 , a ; W ); W − ). (18) Note that DDQN is a model-free algorithm, which guarantees its robustness for different scenarios. Meanwhile, it is an off-policy algorithm which learns from the greedy policy and choose the action according to -greedy algorithm to make a trade-off between exploitation and exploration. The agent will choose actions uniformly from A with a probability of , while choosing the action a = max a q(s, a; W ) which maximizes the Q-value with a probability of (1-). In this paper, we leverage an improved algorithm called decayinggreedy algorithm as shown in [38], so that we can achieve a better explored performance at the beginning and converged performance in the end.

D. Multi-agent observation state
In the multi-agent resource allocation process, each D2D link is modeled as an agent, concurrently making there own decision based on the local observation. Given the current environment state s t , agent D i generates the unique observation z Particularly, the multi-agent learning process can be described as Markov game. The state transition depends on actions taken by all of the agents, i.e., the joint action contributes to the state shift. Apart from the action taken by an agent itself, the actions of other agents can impact the reward of the agent, forming an unstable environment. The nonstationary environment from the view of each agent leads to nonstationary Q-function, making RL hard to converge. The nonstationarity challenge is tackled in [40] with a unique state, which includes view-based positional distribution and shared position information by each vehicle.
The problem is severest when combining with deep learning. DDQN uses experience replay to feed the neural network, while the environment that generated the data in the agent's replay memory is different from the current environment, and the convergence of the learning process is affected. To enable replay memory in MARL, authors in [41] designed a low dimensional fingerprint which includes the information of policy change of other agents. The policy change is highly correlated with iteration times e and the exploration rate . In other words, the observation space for i-th agent can be expressed as z Such fingerprint allows an agent to expect the policy change of other agents, thus improving the stationarity of the environment.

E. Actions and rewards definition
Assuming that orthogonal frequency division multiplexing (OFDM) is applied for the uplink of cellular network, which means cellular users communicate with the BS on disjoint RBs. Each D2D pair can choose one of K RB that is preoccupied with a cellular user. The range of D2D transmit power including A p multiple discrete levels is [0, p D max ]. As the result, the dimension of the action space is equal to A p × K. The actions of all agents form a joint action a t which represents the resource reuse scheme. Initialize the implementation of RIS randomly; 5: Update the large-scale fading channel; 6: for each time slot t do 7: for each D2D agent i do Form the joint action a t and receive reward r t ; 12: Update the small-scale fading channel; 13: for each D2D agent i do 14: Observe z (i) t+1 ; 15: Store transition e t = (z 16: end for 17: end for 18: for each D2D agent i do 19: Replay memory: 20: Sample random minibatch of transitions e j = (z Calculate q target by (18) 22: Perform a gradient descent step on end for 24: end for Reward represents the objective of the optimization. All agents receive the same reward r t according to the joint action a t such that encouraging cooperative behaviors. The reward can be defined as r t = C, if (13b) and (13c) are satisfied; 0, else. (20)

F. Training algorithm
As introduced above, each DDQN at the D2D pairs takes its observation state as the input. Several fully connected layers are leveraged as the hidden layer. During the training and testing phases, the RIS is randomly implemented and updated at the beginning of each episode. One training episode contains several time slots during which the agents interact with the wireless communication environment and store the experience in the training data sets, i.e., replay memories. The details of the training algorithm for the MARL at the D2D pairs is shown in Algorithm 1.

IV. RIS OPTIMIZATION PROCESS
After resource allocation decisions are made by D2D pairs, the resource sharing scheme will be sent to the BS as a part of the input information of the DDQN to optimize the RIS. The optimized RIS position and phase shift information will be broadcast to each D2D pair as shown in Fig. 3. Based on the resource sharing information, the sum rate maximization problem can be formulated as Based on the resource allocation information, a DDQN learning model is proposed at the base station to solve the joint RIS positioning and phase shift problem. Particularly, the RL components are first defined and the DDQN components are then introduced. The proposed algorithm is now explained in detail.

A. RL components definition
In our proposed algorithm, the state contains the position information of D2D users, cellular users, BS and RIS, the resource sharing coefficient ρ, the transmit power of each D2D transmitters, as well as the phase shift Θ. The reward represents whether we encourage or punish an action, so it is defined based on the objective function given in (6). For a successful transmission at iteration t, i.e., the constraints (12b) and (12c) are satisfied, the reward r s can be defined as r s (t) = C(t), where C(t) represents the achievable rate C at iteration t. However, if any of the constraints are not satisfied, the expected QoS cannot be achieved. This kind of action results in a penalty due to energy waste, and we defined the new reward for the transmission failure as The fail reward is to encourage the communication system to improve the SINR which is not satisfied with the requirement. The overall reward can be expressed as

B. Proposed double DQN algorithm for the control of RIS
Leveraging the NN, the DDQN model can find the relationship between the input location information and the corresponding optimized deployment of RIS. The components in DDQN is defined as • Agent: The agent in our DDQN model is the BS. The BS will process the inputs and execute the outputs of DDQN to control RIS. • Input: The DDQN model takes the states S as the input, which includes the position and phase shift information. • Output: The output of the DDQN model is the evaluated Q-value for state-action pairs. The output layer contains |A| units, which represents the number of possible actions. As shown in Fig. 2, two identical networks are set: evaluation network and target network. In the evaluation network, the current state s j is the input information, and the output is the evaluate Q-value for each action. In the target network, the next expected state s j+1 is the input, while the output is the Q-value for each action in the next state. The training process for the DDQN is shown in Section III. By receiving the input information of position information of D2D pairs, cellular users, and the BS from the wireless environment, the RIS controller can train the weights and update NNs to estimate the action-value function. The proposed DDQN algorithm for RIS optimization is shown in Algorithm 2.

C. Computational complexity and communication cost analysis
Generally, floating point operations (FLOPs) is used to measure the computational complexity. For each fully connected layer, the number of FLOPs is [N in + (N in − 1) + 1] × N out , where N in and N out represents the number of neurons. For Algorithm 2 DDQN algorithm for the RIS optimization. Execute multi-agent DQN and perform resource allocation; 5: Update the large-scale fading channel; 6: for each time slot t do 7: Observe s t ; 8: Choose a t ∈ A according to -greedy algorithm; 9: Execute a t and calculate reward r t by (10); 10: Update the small-scale fading channel; 11: Observe s t+1 ; 12: Store transition (s t , a t , r t , s t+1 ) in D; 13: end for 14: if learning begins then 15: Replay memory: 16: Sample random minibatch of transitions (s j , a j , r j , s j+1 ) in D; 17: Calculate q target according to (18); 18: Perform a gradient descent step on (q target − q(s j , a j ; W )) 2 ; 19: Update the target network every g episodes; 20: The proposed algorithm could be trained offline because it is robust to the dynamic environment. Compared to the alternative maximization (AM) approaches [26]- [28] that optimize the resource allocation and RIS configuration by iterations, the proposed trained model only requires a little computational complexity to generate solutions.
Compared with the centralized algorithms that the users need to upload the local information to the BS and receive the optimized control signals from the BS, the proposed algorithm enables users to complete the resource allocation process locally, thus reduce the communication cost significantly. In a nutshell, the proposed DDQN based algorithm outperforms other benchmarks in varying wireless communication environments, where the solution generated by AM approaches via previous iterations may not be applicable for the current environment.

V. NUMERICAL RESULTS
In this section, the performance of the proposed distribute DDQN (D DDQN) based algorithm is evaluated by comparing it with the benchmark algorithms. Assuming that the cellular users are uniformly distributed in a 1000m × 1000m square. The whole area is divided into O = 25 identical squares, where RIS can be installed in any of them. The channel models parameters are listed in TABLE I.
In the proposed MARL structure, each DQN consists of 5-layer fully connected (FC) neural networks with 3 hidden layers. The number of neurons in the three hidden layers are set to 500, 250, and 125, respectively. We apply the rectified linear unit (ReLU) function as the activate function, which is defined as f (x) = max(0, x), while the RMSProp optimizer [42] is applied to train the NNs. For each DQN at D2D users for the resource allocation, the number of training episodes is set as 1500. Note that the trained resource allocation model only needs to be updated when the wireless communication system experience significant changes, thus the resource allocation model is first trained and remains unchanged during the optimization of the RIS. We train the DDQN for RIS optimization for 3000 episodes and the exploration rate decreases from 1 to 0.05 over 2000 episodes linearly. The discount factor γ is set to 0.9.
We compared the proposed D DDQN algorithm with other benchmark algorithms derived from the following schemes. 1) Baseline 1: Baseline 1 is achieved by the optimal resource allocation using exhaustive search from all possible channel assignments and RIS implementations. A. Training Performance of Proposed Algorithms Fig. 4a and Fig. 4b demonstrate the performance of the proposed DDQN training algorithm by comparing it with the DQN algorithm. It is noted that these two figures shows the loss and the reward of the NN at the BS, and the resource allocation process is performed by MARL. Fig. 4a illustrates the loss comparison of the DDQN at the BS. The loss of the DDQN algorithm is lower than the DQN training method during the whole training episodes and can achieve faster convergence since it avoids the overestimation problem caused by the DQN based approach. A lower loss of the proposed algorithm leads to a higher training reward since the estimated maximum value of NN is closer to the practical maximum    value. As shown in Fig. 4b, both average rewards per episode of DDQN and DQN algorithm improve as training continues, while the proposed DDQN approach outperforms the DQN based method. Due to the high training loss at the beginning, the DQN based method achieve lower reward than the DDQN based algorithm.

B. Effectiveness and robustness testing
In the testing phase, we verify the effectiveness of the proposed D DDQN algorithm. The number of testing step is set as 100 episodes and the exploration rate is set as 0 which means the BS always takes the action that has the highest Qvalue. As illustrated in Fig. 5, both algorithms can achieve near-optimal performance, reaching about 95% and 94% of the optimal solution. Note that the MARL achieves less sum rate than the single-agent RL. This is because it only relies on the local information and could reduce the complexity at the BS.
We also test the DDQN based algorithm with other benchmarks in various wireless communication scenarios to verify the robustness of the proposed algorithm. Fig. 6 shows the sum rate performance under various SINR scenarios. It is observed that the sum rates of all schemes monotonically decrease with higher noise power. As expected, C DDQN algorithm and the proposed D DDQN algorithm shows nearoptimal performance. On the other hand, the random scheme for baseline 2 and transitional wireless communication scheme without the enhancement of RIS behave worse than all of the machine learning optimization methods, providing around only 60% of the optimal sum rate. Numerical results confirm the performance gain of the proposed algorithm over non RIS schemes, reaching around 50% improvement of the sum rate performance.  Fig. 7 further demonstrates the sum rate with different number of cellular users. With the increase of accessible channels, the sum rates of all schemes increases. We can observe that the schemes with RIS always outperform without RIS scheme in terms of achieved sum rate, and the gap between the proposed algorithm and the random baseline 2 increases with the number of RBs, which demonstrates the effectiveness of the implementation of RIS. It is also noted that our proposed D DDQN algorithm outperforms the D DQN and random scheme in all the cases.
C. Impact of the number of the RIS elements Fig. 8 demonstrates the influence of the number of RIS elements on the rate performance. It is observed that the sum rate of the considered network increase with a higher number of RIS elements implemented, while the sum rate for no RIS scheme remains unchanged. Compared to the Baseline 1 with extremely high complexity, the proposed design can achieves 93% and 95% of the sum rate performance of the upperbound algorithm when N is 16 and 96, respectively. The suboptimal performance of the proposed algorithm verifies that the undesired channel interference can be effectively suppressed by adequate RIS implement. However, the performance improvement slows down as the number of elements becomes larger, which is caused by the servere interference between cellular networks and D2D communications.

VI. CONCLUSION
In this paper, we consider the joint optimization of resource allocation and RIS implementation to maximize the sum rate of the D2D network. To solve the non-convex problem, a novel MARL structure is proposed to perform the channel assignment and RIS optimization. We decouple the joint optimization into sub-problems and reduce the computational pressure at the central BS by decentralized resource allocation. Leveraging the DDQN algorithm, the RIS optimization is performed at the BS centrally. The simulation results demonstrate that the proposed algorithm can achieve near-optimal performance and reduce the computational pressure at the BS significantly. The proposed algorithm outperforms other baseline algorithms under different wireless communication scenarios, verifying the effectiveness and robustness of the DDQN scheme.