{'fraction_non_alphanumeric': 0.040326510721247565, 'fraction_numerical': 0.005847953216374269, 'mean_word_length': 4.580217539089055, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Hybrid Offline-Online Design for Reconfigurable Intelligent Surface Aided UAV Communication

This letter considers the reconfigurable intelligent surface (RIS)-aided unmanned aerial vehicle (UAV) communication systems in urban areas under the general Rician fading channel. A hybrid offline-online design is proposed to improve the system performance by leveraging both the statistical channel state information (S-CSI) and instantaneous channel state information (I-CSI). For the offline phase, we aim to maximize the expected average achievable rate based on the S-CSI by jointly optimizing the RIS's phase-shift and UAV trajectory. The formulated stochastic optimization problem is difficult to solve due to its non-convexity. To tackle this problem, we propose an efficient algorithm by leveraging the stochastic successive convex approximation (SSCA) techniques. For the online phase, the UAV adaptively adjusts the transmit beamforming and user scheduling according to the effective I-CSI. Numerical results verify that the proposed hybrid design performs better than various bechmark schemes, and also demonstrate a favorable trade-off between system performance and CSI overhead.


I. INTRODUCTION
For the sixth-generation (6G) wireless communications, the new concept of reconfigurable radio environment was proposed to overcome the random and time-varying channels, which can be achieved by two emerging techniques: unmanned aerial vehicle (UAV) and reconfigurable intelligent surface (RIS) [1]. Specifically, UAVs operating at low altitudes are with high flexibility and maneuverability, and its trajectory can be designed to create favorable communication channels with ground users. Besides, RIS is a programmable metamaterial surface consisting of many low-cost passive reflecting elements, changing the propagation direction of electromagnetic waves by adjusting the phase shifts of elements [2].
Recently, the RIS technique has been integrated into UAV communication systems to reconfigure the propagation environments of air-to-ground channels, thus extending the communication service coverage and enhancing the communication quality [3]- [6]. In [3], the authors studied the maximum average achievable rate problem in RIS-aided UAV communication systems by jointly optimizing the UAV trajectory and RIS's phase-shift. In [4], the RIS was also applied to aid the full-duplex UAV communication systems that considered both uplink and downlink transmissions. The authors in [5] studied the ultrahigh reliability problem for internet of things devices communicating by both UAV relay and RIS. Besides, the authors in [6] considered the fair downlink communications for RIS-UAV enabled mobile vehicles.
It is worth pointing out that the aforementioned studies adopted the deterministic line-of-sight (LoS)-dominant channel model, which is usually a valid assumption for rural areas. As a result, these works considered the deterministic optimization problems and proposed algorithms working in an offline manner. However, empirical studies showed that the Rician factor, which is the ratio of the power of the deterministic LoS component to the stochastic NLoS component, may drop to −5 dB in the low altitude platform of urban areas [7]. In these scenarios, the existing offline design may suffer considerable rate performance loss, since it cannot adapt to the random and dynamic UAV-ground channel states [8]. Although some recent works have adopted the Rician fading channel model in RIS-aided UAV communication systems (e.g., [9]- [11]), they still followed the offline design based on the deterministic optimization, which is not suitable for the environments with relatively small Rician factor. Note that the deep Q-network (DQN) based online design was utilized in RIS-aided UAV communication systems, such as in [12]. However, these works mainly focused on the obstacle avoidance and random user movement instead of the uncertain channel states. Besides, they need to obtain the full instantaneous channel state information (I-CSI) between UAV, RIS and users in real-time, which is quite challenging due to the mobility of the UAV and the passivity of the RIS [10]. Thus, it is important to investigate how to improve the system performance for RISaided UAV communication systems under the general Rician fading channel model with limited CSI overhead.
Motivated by the above, we propose an efficient algorithm for the multi-user RIS-aided UAV communication systems in both offline and online phases, 1 which is easier to implement in the urban areas. Specifically, in the offline phase, our goal is to maximize the expected average achievable rate from all the users according to the statistical CSI (S-CSI). To tackle this problem, we propose an iterative algorithm based on the stochastic successive convex approximation (SSCA) techniques [16]. For the online phase, with the fixed RIS's phase-shift and UAV trajectory, the transmit beamforming and user scheduling are dynamically designed to accommodate the effective I-CSI, and the computational complexity of the online design facilitates the practical implementation. Our simulation results demonstrate that the proposed hybrid design achieves a better system performance with limited CSI overhead.
II. SYSTEM MODEL We consider a RIS-aided UAV communication downlink transmission system consisting of a rotary-wing UAV equipped with a uniform linear array (ULA) with N t elements , K single-antenna ground users, and a RIS mounted on a building.
The UAV acts as an aerial base station (BS) and flies at a fixed altitude z F within a given period T to meet the communication service requirements of users. To facilitate the design of the UAV trajectory, T is equally divided into N time slots with step size δ t , i.e., T = N δ t . Therefore, the UAV trajectory can be approximated by the sequence q[n] = [x[n], y[n]] T , n ∈ N = {1, · · · , N }. With a given maximum UAV speed v max , N can be chosen properly such that the time for UAV location changing within δ t can be negligible. Accordingly, the UAV trajectory should satisfy the constraints as where D = v max δ t is the maximum horizontal distance that the UAV can move within a single time slot, and q 0 and q F denote the UAV's initial and final horizontal locations, respectively. The RIS is deployed to enhance the system performance, which is equipped with a uniform rectangular array ( Following [9], [10], we assume the Rician fading channel model for all communication links. Specifically, the channel of the RIS-ground user links can be modeled as where d RG,k [n] is the distance between the RIS and ground user k in the nth time slot, ρ is the path loss at the reference distance D 0 = 1m, and z r,k [n] ∈ C M×1 has circularly symmetric complex Gaussian (CSCG) entries with zero mean and unit variance accounting for small-scale fading in the nth time slot. Moreover, α, β RG and z r,k [n] denote the path loss exponent, Rician factor and deterministic component of the RIS-ground user links in the nth time slot, respectively. The UAV-RIS and UAV-ground user links in the nth time slot can be generated with a similar procedure i.e., where Z[n] and z d,k [n] denote the deterministic components in the nth time slot, and Z[n] ∈ C M×Nt and z d,k [n] ∈ C Nt×1 are the corresponding small-scale fading components in the nth time slot similarly to z r,k [n]. We adopt the time-division multiple access (TDMA) manner for the users, which means that the UAV can only communicate with only one user at one time slot. Define a binary variable a k [n], which indicates that user k is served by the UAV in the nth time slot if a k [n] = 1, and a k [n] = 0, otherwise. Then, we have the following scheduling constraints If the UAV communicates with user k in the nth time slot, the achievable rate (bits/second/Hertz) can be expressed as where w[n] ∈ C Nt×1 is the transmit beamforming of the UAV in the nth time slot, and σ 2 is the noise variance.

III. PROBLEM FORMULATION
Our objective is to maximize the average achievable rate via jointly optimizing the transmit beamforming , n ∈ N , k ∈ K}, and the UAV trajectory Q {q[n], n ∈ N } over the entire N time slots. Thus, the optimization problem can be expressed as where P is the maximum transmit power of the UAV. The optimal solution to problem (P A ) is difficult to obtain, since it is impossible to acquire the accurate I- } at all possible UAV locations in the region of interest before the flight. Considering this difficulty in practice, we propose a hybrid offline-online design for RIS-aided UAV communication systems, consisting of the following two phases.
Offline phase: Prior to the UAV's flight, we offline design the UAV trajectory and RIS's phase-shift to maximize the expected average achievable rate based on the Rician fading channel model and user's location. Therefore, the resulting RIS's phase-shift and the UAV trajectory are statistically favorable solutions. The optimization problem of the offline phase is formulated as where E is the expectation operator.
Online phase: Since the number of transmit antennas at the UAV is usually much smaller than the elements of the RIS, the usually has a much smaller dimension than the full I-CSI F[n] [14]. Thus, we only estimate the effective I-CSI in real-time along its flight by existing methods [15], and the number of channel coefficients required in each time slot can be reduced from Then, the transmit beamforming and the user scheduling can be adjusted online to cater to the random environments. The optimization problem of the online phase in the nth time slot is formulated as , k ∈ K} is the user scheduling in the nth time slot.

A. Proposed Offline Design
For the offline design, we develop an alternating optimization algorithm to tackle the non-convex problem (P B ). Specifically, problem (P B ) is partitioned into four subproblems: the optimization of the transmit beamforming, the RIS's phaseshift, the user scheduling, and the UAV trajectory.
1) Transmit Beamforming Optimization: It is well-known that transmit beamforming as the maximum ratio transmission (MRT) is optimal i.e., .
By substituting w opt [n] to (6), the average achievable rate of the user k in the nth time slot can be simplified as T for the kth user in the nth time slot. Thus, with given A and Q, the optimization problem of Φ in the nth time slot can be written as where H k [n] = diag h H r,k [n] G[n] . Problem (P B1 ) is difficult to solve, since the objective function is non-convex and has expectation operation. To tackle this difficulty, we apply the SSCA framework in [16] and optimize the problem iteratively. The details are as follows.
First, we approximate the gradient of (P B1 ) with I channel samples in one batch. Specifically, at the lth iteration, Fi[n] = {Gi[n], h r,k,i [n], h d,k,i [n]} i={1,··· ,I} are randomly generated according to the S-CSI. The realization of the objective function in (P B1 ) is given by According to the chain rule, the gradient in one realization is given by where ∇f (x) is the gradient vector of function f (x) with respect to vector x, θ * k [n] is the conjugate of θ k [n], • denotes the Hadamard product, and Re{·} is the real part of a complex number. Thus, the stochastic gradient of problem (P B1 ) at the lth iteration is updated by where ∇f k ϕ (l) [n] is the stochastic gradient in one batch, and ζ = l −ν , 0.5 ≤ ν ≤ 1.
Second, we update the surrogate function f k ϕ k , ϕ (l) k [n], which is constructed by the second order Taylor expansion and can be viewed as a concave approximation of f k (ϕ k )[n] as where denotes the inner product operation, and τ > 0 is a constant representing the learning rate. Therefore, we have the following convex optimization problemφ and the closed-form optimal solution of (17) iŝ Finally, ϕ k [n] at the lth iteration is updated by where ξ = l −µ , and δ < µ ≤ 1 to guarantee convergence [16]. Thus, the optimal ϕ[n] is given by 3) User Scheduling Optimization: With given Φ and Q, the optimization problem of A can be expressed as We use a batch of randomly generated channel samples to deal with the expectation operation in the objective function by applying the sample average approximation method in [2]. Thus, (P B2 ) can be approximately solved by (P B2.1 ).
Problem (P B2.1 ) is a linear optimization problem by relaxing the binary variable. Thus, it can be efficiently solved by using standard optimization solvers such as CVX [17]. The details of this method can be found in [11], and are omitted here for brevity.

4) UAV Trajectory Design:
With given A and Φ, the optimization problem of Q can be expressed as By applying the same procedure as in the user scheduling optimization, problem (P B3 ) can be approximately solved by problem (P B3.1 ).
By taking the first-order Taylor expansion at given points as its lower bound. Specifically, the first-order Taylor expansions of R k [n], u 2 k [n] and v 2 [n] are respectively given by where D (j) As such, problem (P B3.2 ) can be approximated as (1).
Problem (P B3.3 ) is a convex optimization problem, and thus can be solved by the CVX.
Algorithm 1 Proposed algorithm for solving problem (P A ).

B. Proposed Online Design
In the online phase, the transmit beamforming and user scheduling are optimized to accommodate the real-time channels by using the effective I-CSI h k [n] in each time slot.
1) Transmit Beamforming Optimization: Similar to the transmit beamforming optimization in the offline phase, when the UAV communicates with the user k in the nth time slot, the optimized transmit beamforming can be expressed as Problem (P C1 ) can be optimized as the same algorithm of the user scheduling optimization in the offline phase.

C. Overall Algorithm
The overall algorithm for solving (P A ) in the offline and online phases is summarized in Algorithm 1.     1 shows the UAV trajectories by various algorithms with T = 100 s. We observe that, when the Rician factor of the proposed hybrid design is large (β = 10 dB), the UAV tries to approach each of the users in its route, but keeps static nearby the RIS as long as possible. This is because the UAV enjoys the gain provided by the RIS and users. However, when the Rician factor is small (β = −5 dB), the UAV tries to approach each of all the four users and hovers above them instead of the RIS. This is because the signals from different transmission paths can not be aligned at users and the gain of the RIS deteriorates severely, thus the UAV flies to the users to obtain the channel gains of direct links.
In Fig. 2, we show the average achievable rate for different schemes versus β with T = 100 s. It is observed that the average achievable rate increases with the Rician factor of all the schemes, because the related channels become more deterministic, and a larger proportion of the LoS components can be obtained to improve the average achievable rate. It is also observed that the hybrid design scheme outperforms the DCM scheme, especially when β is small, since the hybrid design fully exploits the S-CSI and I-CSI. This indicates that the hybrid design is a better choice for this system, especially in the complex and random urban environments. Besides, the performance gap between the hybrid design scheme and heuristic trajectory scheme reduces as β increase, because the UAV prefers to be close to the RIS when β is large.
In Fig. 3, we show the average achievable rate for different schemes versus T with β = 5 dB. We observe that the achievable rate increases with T , since the UAV can fly to more favorable hovering locations to boost the system performance. Besides, the proposed hybrid design exceeds the heuristic trajectory and random phase-shift schemes. This shows that, by optimizing the UAV trajectory and RIS's phaseshift, the system performance can be effectively improved.

VI. CONCLUSIONS
In this letter, we investigated the RIS-aided UAV communication systems under the general Rician fading channel model and proposed the hybrid offline-online design to solve the maximal average achievable rate problem efficiently. Specifically, the offline design determines the UAV trajectory and RIS's phase-shift based on the S-CSI. The online design adjusts the user scheduling and transmit beamforming based on the effective I-CSI. Simulation results demonstrate the effectiveness of the proposed hybrid design.