{'fraction_non_alphanumeric': 0.031198630136986302, 'fraction_numerical': 0.006335616438356164, 'mean_word_length': 4.504429783223374, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multi-UAV Mobile Edge Computing and Path Planning Platform based on Reinforcement Learning

Unmanned Aerial vehicles (UAVs) are widely used as network processors in mobile networks, but more recently, UAVs have been used in Mobile Edge Computing as mobile servers. However, there are significant challenges to use UAVs in complex environments with obstacles and cooperation between UAVs. We introduce a new multi-UAV Mobile Edge Computing platform, which aims to provide better Quality-of-Service and path planning based on reinforcement learning to address these issues. The contributions of our work include: 1) optimizing the quality of service for mobile edge computing and path planning in the same reinforcement learning framework; 2) using a sigmoid-like function to depict the terminal users' demand to ensure a higher quality of service; 3) applying synthetic considerations of the terminal users' demand, risk and geometric distance in reinforcement learning reward matrix to ensure the quality of service, risk avoidance, and the cost-savings. Simulations have shown the effectiveness and feasibility of our platform, which can help advance related researches.


I. INTRODUCTION
M OBILE data processing technology is experiencing a growing demand in the communication market. New technologies like 5G are emerging to accelerate its development. However, the demands of terminal users in uncertain environments and extreme situations have never been perfectly satisfied as the calculations and services are often hard to reach from base stations. As a result, mobile edge computing comes out as one of the fastest-growing topics in telecommunications in the past few decades [1].
Mobile edge computing is a concept that integrates the capabilities of the network, computing, storage, and intelligence services on the edge of the network physically close to the data source. In a typical mobile edge computing scenario, terminal users are served by edge servers with high computing power [2], [3]. The effectiveness of mobile edge computing is measured by the Quality of Service (QoS) of each terminal user. The higher the QoS, the more efficient a terminal user's demand is satisfied or served.
Unmanned Aerial Vehicles (UAVs) have become ideal servers for mobile edge computing that assures QoS, improving stability, reliability, and calculating efficiency through research and development investment [4]- [6] . They are also flexible and cost-effective due to their small size [7], [8]. Therefore, a UAV can move flexibly from terminal user to terminal user and conduct highly efficient calculating services to improve QoS.
UAV-mounted mobile edge computing remains challenging due to the complexity of the working environment, the uncertainty of the distribution of terminal users, and the limitations of the UAV's energy [1]. Therefore, path planning plays an indispensable role when using UAVs for mobile edge computing to handle these issues. For example, [9] proposed a multi-agent algorithm to determine the optimal path of UAVs based on Q-learning. An echo state network (ESN) based prediction algorithm is also proposed for predicting the future movement of terminal users. Liu et al. [10] built a model to evaluate QoS and proposed an algorithm to maximize the reward during the planning process.
However, most of the existing research focuses on finding the optimal path under a specified mission or treating the planning process as a simple greedy strategy without a continuously improving process. This prevents the UAVs' adaption to the changing environment, and the planning can easily fall into a local optimum. Furthermore, previous works on UAV-mounted mobile edge computing seldom consider risk avoidance or collisions between UAVs as this is impractical in real environments.
To find an adaptive global optimum solution for different tasks, [11], [12] proves that Reinforcement Learning (RL) is effective. [13] dramatically improved QoS by introducing RL and regards the path planning problem as an optimization problem with constraints from the environment. Compared with traditional path planning methods such as the A* algorithm and RRT, RL is more flexible for the following reasons: 1) In the mobile edge computing scenario, the terminal user's residual demand dynamically changes, which requires realtime policy updating. However, traditional methods cannot work efficiently in such time-varying scenarios; 2) There are both obstacles and terminal users on the map, which requires not only obstacle avoidance but also task allocation, and thus it is difficult for those algorithms that only consider geometrical constraints to handle this issue; 3) In RL, elements in the environment can be depicted uniformly by the cost function, thus making it possible to flexibly adjust the policies by changing coefficients in the cost function according to different mission' needs. As a result, RL is more adaptive to various scenarios and is more suitable to be a base for building the mobile edge computing platform.
Motivated by the reasons mentioned above, we propose a platform to advance research in UAV-mounted mobile edge computing by building a unified framework with path planning algorithms based on RL. The main contributions of this paper are summarized as follows: • First, we provide a novel framework in which UAVmounted mobile edge computing and path planning are combined based on RL by considering the geometric distance, risk, and terminal users' demand in a single cost matrix. • Second, we investigate multi-UAV collaboration in the mobile edge computing scenario. Geometric and terminal user's information are shared among UAVs, thus ensuring cost-saving and obstacle avoidance. • Third, we introduce an efficient way of depicting the terminal users' demand for achieving a higher QoS. Compared with the traditional linear demand function, the sigmoid-like function enables better task allocation. • Fourth, we perform extensive experiments to test the proposed platform and evaluate different coefficients in the cost function. Results have shown the effectiveness and feasibility of our approach.
The rest of this paper is organized as follows. First, related work is presented in Section II. Then in Section III, we provide a detailed description of the UAV-mounted mobile edge computing platform. Finally, the effectiveness and feasibility of the proposed platform are validated by simulations in Section IV, and conclusions are presented in Section V.

II. RELATED WORK
We conduct our literature review to cover mobile edge computing, path planning, and their combination.

A. Mobile edge computing
Mobile edge computing has attracted increased interest and is becoming one of the hottest topics in edge computing. For example, [14] develops a taxonomy of mobile edge computing applications and use cases. In addition, there some representative reviews: [1] presents related concepts and technologies, architectures, advantages, and typical scenarios of mobile edge computing, [3] explains architecture and computation offloading, and [2] investigates mobile edge computing communications. These works demonstrate the potential of this field.
While the concept of mobile edge computing has been seen in the literature before, it remains an open problem. According to [3], the distribution and management of mobile edge computing resources is a key requirement for ensuring the QoS of terminal users. When servers are moving dynamically, the system benefits from flexibility but at the same time becomes more complicated, which deteriorates the incapability of most arrangement methods.

B. Path planning
Kim et al. [15] introduce a path planning algorithm for dynamic environments where small UAVs are used as relay nodes in a network of naval vessels. The motion estimates of the vessels and the states of the UAVs are taken as input to generalize the strategy. Instead of optimizing a centralized system, the approach exploits a fully decentralized non-linear model predictive control concept. To emphasize the cooperation between UAVs, Zhang et al. [16] propose the Cooperative and Geometric Learning Algorithm (CGLA) designed for path planning based on the cooperation of multiple UAVs. CGLA introduces a weight matrix based on geometric distance and integral risk information to guide the movement of UAVs. The weight matrix can be efficiently calculated and updated, making the system lighter than systems based on methods such as neural networks and guarantees real-time path planning. We note that there is a relatively low requirement for computing power in CGLA reinforcement learning, and this makes it a suitable approach for UAV-mounted mobile edge computing.

C. Combination of mobile edge computing and path planning
Previous works introduce successive convex approximation (SCA) to combine mobile edge computing and path planning. Jeong et al. [17] leverage SCA strategies to calculate the path of UAVs under latency and UAV's energy budget constraints. [18] investigates a scenario where the UAV offloads its computation tasks to multiple ground stations along its trajectory. The authors exploit alternating optimization and SCA techniques to design the UAVs trajectory to minimize the mission completion time. However, these works do not involve a cooperative mechanism between UAVs. In addition, these approaches have a limitation when the environment is unknown in advance. The learning-based algorithms also attract the attention of path planning in UAV-mounted mobile edge computing. For example, [19] investigates a joint task scheduling and resource allocation approach in a space-airground integrated network based on policy gradients and actorcritic methods. [20] applies a deterministic policy gradient algorithm to maximize the uplink sum rate in the UAV-aided cellular network with multiple ground users.
Inspired by these works, we provide an organic combination of mobile edge computing and path planning by building an open-source platform for the UAV-mounted mobile edge computing networks. Simulation results have shown the feasibility and flexibility of our platform.

III. MULTI-UAV MOBILE EDGE COMPUTING AND PATH PLANNING BASED ON REINFORCEMENT LEARNING
In reinforcement learning, the agent's goal is to find an optimal strategy under each scenario to obtain the maximum expected reward. In our platform, the mobile network processor UAV is the agent who constantly learns from the environment. At each time slot, a UAV chooses a planning strategy to achieve the best possible reward according to its surroundings. After the UAV moves, the surrounding changes and positive or negative feedback is provided to the UAV in the form of a reward matrix A given the factors of risk, geometric distance, and terminal users' demand. The UAV then learns from the surroundings through a stochastic iterative cost matrix G generated from A and selects a strategy-a path toward the target. G can be considered as the memory of each agent, which is intensified and 'trained' through each episode of the planning process.

A. Environment modeling
This paper considers the UAV's collision avoidance and terminal users' demand on the same platform. The surroundings contain two basic elements of obstacles and terminal users. First, obstacles vary in shape, positions, and risk levels, which include buildings, cars, or mountains in a real environment. Second, we assume the obstacles comply with a Gaussian distribution but have different variances σ used to calculate their risk exposure probability.
For n independent obstacles in the map, giving the ith obstacle position O i = (X i , Y i ), the risk r i (x, y) indicates the risk from O i at the point (x, y) and can be defined as Considering all n obstacles in the map, the overall risk to a point (x, y) in the risk exposure probability matrix can be described as (2) The exposed risk from any point p to any point q on the map is the integral risk of R(x, y) for any (x, y) on C, where C is the linear path from p to q: Second, for serving terminal users, we assume that each terminal user has an initial demand d 0 j for the UAVs to process. We also assume the demand can only be served by UAVs within a constant service radius because the UAVs have limited capability for detecting demand signals beyond a certain distance. Therefore, the service area is denoted s(p j , ), where p j is the position of T U j , and is the service radius, as shown in Figure 1.
When a UAV enters the service range of T U j , the service for T U j begins. The remaining demand of T U j will decrease at a constant speed τ per unit time per UAV. We can easily infer that a terminal user with a greater demand needs more time to be served, and the longer a UAV remains in the service range of T U j , the more service can be provided for T U j . d j changes over time t k time served by U AV k as The correlation between the terminal users' demand and UAVs' demand detection should have a non-linear relationship to improve system performance. With reference to [10] and [21], a sigmoid-like function can help enhance strong signals and abate weak signals. Thus, we adopt a sigmoid-like demand detection function U (d j ) ∈ (0, 1] to describe the correlation between real demand and detected demand as where η and β are controlling variables. From 2(a), U (d j ) first increases steeply as the demand rises and becomes steady when the demand is sufficiently great. Thus, (5) can encourage a UAV to focus on terminal users with greater unserved demand and prevent it from serving any terminal user over a long time, thus improving QoS.
Generally, a Sigmoid-like function is an increasing function with an inflection point x 0 , which follows [21]. Functions with such form satisfy the following properties: Property 1: For any x > 0 in U (x), the function is valid only when η ∈ (1, ∞), β ∈ (0, ∞) The proof is given in Proof 1 in the appendix. Property 2: η controls the slope and centrality of the curve, pivots through an inflection point (1, 1−e − 1 1+β ). β controls the horizontal movement of the curve. As a result, the intersection point can be moved vertically by changing β.
The proof is given in Proof 2 in the appendix. As shown in Figure 2(b), U (x) has an inflection point when x = 1. The curve becomes steeper when η increases and moves vertically downward when β increases. Therefore, η and β are constant variables that affect QoS. The evidence will be provided in part IV.
To simplify the model, we assume the demands are linearly accumulated. For a UAV at a point p in the map, with a detection range , the detected demand is the linear accumulation of terminal users' demand within circle area s(p, ):

B. The Reward matrix
A reward matrix is introduced for UAVs to learn and adapt to find the optimal path. The reward matrix is designed to measure the reward or punishment from any point towards any other points on the map given the factors of risk, geometric distance, and terminal users' demand.
In our platform, the map is represented as a lattice of N ×N , and the reward A pi,pr between any point p i and p r in the map is defined as where d pi,pr is the geometric distance between p i and p r . The second term in the equation denotes the risk detected from p i to p r or vice versa, which means the larger the risk detected, the larger the cost or punishment. The last term in the equation is the overall demand detected at p i , as p i is the current position of UAV. The larger the demand detected, the smaller the punishment, or the larger the reward.
For each point p r , r ∈ 1, 2, · · · , N 2 in the map, a reward matrix A pr consists of points A pi,pr , i ∈ 1, 2, · · · , N 2 is generated with regard to all points p i on the map. K and M reflect the tolerance of risk and priority of service, which affects the strategy for path planning. When applied in real situations, K and M can be adjusted according to the mission's requirements. For instance, if K is set to a relatively high value, the UAVs will tend to stay away from obstacles even though this leads to a longer path length.
An obstacle observation radius is introduced for each UAV to fit real situations. When an obstacle enters the observed area of a UAV, the UAV detects the obstacle and obtains risk information. Only the observed obstacles will count as a risk when calculating the weight matrix. The highlighted color in Figure 3 shows the observed risk calculated by (1) and (2) for a UAV in position P with an observation radius R.  Figure 3 and the overall terminal user's demand. According to (7), the terminal user's demand decreases the cost, and the observed risk adds the cost. In Figure 4, we can infer that places with more terminal user demand have lower values (dark color), and those with more obstacles have higher values (light color).

C. Cost matrix
During the path planning process, a cost matrix G is introduced for a UAV to obtain a preliminary optimal path to the destination. The generation of the cost matrix is obtained using an iterative process. After several iterations, the cost matrix will converge [16] and remain steady. The update mechanism of the cost matrix in a map with N ×N computing nodes is described as: 1) Initialize G: initialize the cost matrix G 0 . Assign the value 0 to the target point and the value ∞ to all other points. 2) Update the cost matrix G. Randomly choose a position p r from the map. For each point p i in the map, update the point value in G by comparing the current value with the revised value considering the reward matrix: 3) Repeat step 2) until reaching the maximum number of iterations. After G is produced, an ordinal sequence of points is generated as a preliminary P ath for the UAV to follow. Each UAV has its own P ath. Points with the lowest cost in G are constantly added. The generation of P ath is described as follows: 1) Initialize P ath as an empty list.
2) Add p i with the lowest value in G to P ath, then assign G pi to ∞. 3) Repeat step 2) until reaching the target point or reaching a maximum length. We note that the elements in P ath are in ascending order of cost. The process of calculating G and generating P ath together form the function Planning in Algorithm 1.

D. UAVs Movement
Each UAV is considered an agent in the RL process. They are therefore assigned a memory D i and a cost matrix G i . D i stores map information and serves as the "eyes" and memory of the agent, and G i serves as the "brain" of the agent. The agent produces a learned result P ath i to complete each episode. All UAVs in the system move in sequence to realize information sharing. This can result in a slight time delay in real situations. When one UAV moves, the other UAVs would be treated as obstacles. Algorithm 1 describes the process. For the i th UAV (U AV i ), after moving one step according to P ath i , ScanEnv is performed. In ScanEnv, U AV i scans the circle area s(pos i , R) where R is the observation radius. This is used to decide whether further planning is needed. If new obstacles, including other UAVs, are observed, ObstacleF ound will be set to T rue, and memory D i will be updated. Then, the weight matrix and G i will be altered, and P ath i will be recalculated. If the surroundings remain unchanged, U AV i continues to Move according to P ath i . In Move, U AV i moves a distance of StepLength along the direction of the vector starting from pos i and ending at P ath i [1], if the distance between pos i and P ath i [1] is smaller than StepLength, the UAV will move directly to P ath i [1]. Move returns the new position of U AV i . At the end of one loop, the remaining demands of all TUs within the service area of U AV i are updated according to (4).

IV. SIMULATION AND DISCUSSION
This section first illustrates the UAV dynamic planning process and discusses the influence of the parameters K and // Remove outdated information from D i because pos j has changed in last loop 10: for j in U AV num and j! = i do 11: delete pos j from memory D i 12: end for 13: ObstacleF ound ← ScanEnv(pos i , R) 14: if ObstacleF ound then  (7) on the planning results. Second, we demonstrate the efficiency of the sigmoid demand function (4) by comparing it with a linear demand function. Finally, we compare our algorithm with a commonly used algorithm A*, which shows that our planning algorithms achieve a much better result in terms of QoS.
To demonstrate our algorithm, we have made several simplifying assumptions: • The site is abstracted to grids so that objects, including obstacles, terminal users, and UAVs, are aligned with the grid. • All object information is stored in databases for UAVs to 'scan,' while in real situations, the scanning process could be realized by perception algorithms based on sensors like cameras or radars. • The variation in speed is realized through adjusting StepLength. In our proposed algorithm, each UAV takes 1 unit of time to move one StepLength. If the distance between a UAV's current position and its planned position is smaller than StepLength, the UAV will move to the planned position instead of moving one StepLength. In practical situations, the calculation distance in a map could be set to smaller than one StepLength, which would require speed variations. However, we are assuming a 2D planning scenario. The altitude of the UAV in a 3D environment is not considered.

A. Path Planning for Multiple UAVs
In this simulation, we set K = 20, M = 1, η = 2, β = 8, , R = 0.2. K, M is decided by users of the platform with different needs, while other parameters are determined by practical conditions and the UAVs' capabilities. Ten obstacles are given random positions with random variances σ i > 0, i = 0, 1, 2, · · · , 10. Six terminal users are assigned random demands d j ∈ [0, 10], j = 0, 1, 2, · · · , 6. In a real situation, the terminal user's demand could be a real-time variable.
The planning process of the three UAVs is shown in Figure  5. Note that: • The point with the black cross marks the target point for all UAVs. All UAVs are tasked to deliver service to terminal users on the map and fly to the target point for each mission. • The red dots show the existence and the amount of the terminal users' demand with a service radius . The red dots shrink while terminal users are being served, representing a reduction in the remaining demand. As can be seen from the overlap area of the service radius, the demands are cumulative. From Figure 5, we can infer that UAVs can choose a lowrisk path to serve each terminal user in a complex environment based on our platform. Terminal users with higher demand are more attractive to UAVs. After the demand is reduced, UAVs will change direction for other demand-high areas. Meanwhile, information sharing is effective in avoiding collisions between UAVs during planning.

B. The Evaluation of M
The parameter M decides the priority of serving the terminal users and therefore controls the QoS. As shown in Figure  6, given a fixed K, in the scenarios with larger M , UAVs are prone to meet more terminal users' demands but take more risk and sacrifice path length (i.e., energy) to do so. On the contrary, UAVs in the scenarios with smaller M fail to serve all terminal users but save more energy on a shorter path.
The same results are also obtained by numerical simulation. To compare the service rate, we define As shown in Figure 7, when M increases, QoS and average risk increase accordingly.
By changing M , our platform can meet the requirements of different missions with different service needs. QoS and risk can be flexibly balanced.
C. The Evaluation of K Similar to parameter M , altering the parameter K can make the algorithm more flexible in different environments. K controls the tolerance of risk. UAVs set with higher K in the result tend to sacrifice energy cost to avoid risk and thereby influence QoS. Figure 8 compares the path planning results with different K. When K increases, a higher risk avoidance strategy is adopted, and the UAV follows a path to avoid obstacles at all costs instead of flying through narrow tunnels to serve the terminal users. Numerical results are shown in Figure 9 to better illustrate the algorithm. When K increases, path length and path risk move in opposite directions.

D. Comparison of the sigmoid demand function and linear demand function
According to [10], a sigmoid processed demand can improve system performance. To verify the effectiveness of (5), we performed experiments separately with the sigmoid demand function j∈s(p, ) U (d j ) and the linear demand function j∈s(p, ) d j where d j ∈ [0, 1]. Results show that the former gives higher QoS with the same experimental conditions and leads to higher service speed with the same QoS. Figure 10 shows that the sigmoid demand function assures a higher QoS than the linear demand function.  Table I compares the service completion time of each terminal user with the two schemes. Accordingly, the sigmoid demand function leads to a higher terminal user service speed with the same QoS. This is because U (d j ) > d j in the early period, as shown in (7) and (8). The UAVs are attracted to the terminal users earlier and thus finish the service more quickly.

E. Comparison of our algorithm with the A* algorithm
The A* algorithm is widely used as a baseline in various path planning scenarios. We compare our proposed algorithm with A* in the multiple UAV mobile edge computing environment.
Based on the A* algorithm, at each step of path planning, a UAV can choose among one of a fixed number of equally distributed directions to move one unit step. In our experiments, we set up eight directions for UAVs and thus have eight candidate nodes p i for a UAV to choose at each step. Considering the path length and risk as a cost and the demand of the terminal users as a reward, we formulate the weight function F i at each point p i in the A* algorithm as where d pi,pt is the Euclidean distance between candidate point p i and the target point p t . R pi and U (d j ) are defined in the previous equations. In general, Figure 12(a) shows the A* algorithm fails to  perform effective service when terminal users are surrounded by obstacles. However, with the proposed algorithm ( Figure  12(b)), the UAVs manage to serve the terminal users while avoiding risk.
QoS is the most important index in a mobile edge computing mission. As shown in Table II and Table III , the proposed algorithm can achieve a higher QoS compared to the A* algorithm. Furthermore, our algorithm can react flexibly to the change of K and M , thus adjust priorities to path length and average risk while assuring QoS. For example, with the proposed algorithm (K = 50, M = 0.5), all three indices are better than using the A* algorithm (K = 0.5, M = 0.5).
A deadlock means that the mission can never come to an end because one or more UAVs are stuck in the current surrounding and loop infinitely. Our experiments show that the A* algorithm can easily fall into a deadlock with parameter K above a certain level. The results are illustrated in Table III  and Figure 13. Based on the above comparisons, our RL platform performs better not only on environment adaptation and high QoS assurance but also on algorithm reliability and high mission completeness. The effectiveness and feasibility of our method are clear.

V. CONCLUSION
This paper develops the first multi-UAV mobile edge computing and path planning platform where UAVs serve terminal users as mobile network processors based on reinforcement learning. We implemented our platform with 1) a Quality-of-Service (QoS) for each terminal user, 2) maximum collision avoidance with minimum risk, and 3) cooperation between UAVs. The simulations and experiments are provided to show the efficiency and usability of the platform, which can be a useful baseline for mobile edge computing.
If η <= 1, then x <= ηβ 1−η , which contradicts the domain of x, thus we have η > 1 Proof 2: We have It means as η increases, U (x) decreases when x < 1 and increases when x > 1, and the slope of the curve increases accordingly. When x = 1, dU dη = 0 for any η, it creates an inflection point that is not affected by changing η.
We also have which means when β increases, the sigmoid demand curve drops vertically to the inflection point. David Doermann is a Professor of Empire Innovation at the University at Buffalo (UB) and the Director of the University at Buffalo Artificial Intelligence Institute. Prior to coming to UB, he was a program manager at the Defense Advanced Research Projects Agency (DARPA), where he developed, selected, and oversaw research and transition funding in computer vision, human language technologies, and voice analytics. From 1993 to 2018, David was a member of the research faculty at the University of Maryland, College Park. David has over 250 publications in conferences and journals, is a fellow of the IEEE and IAPR, has numerous awards, including an honorary doctorate from the University of Oulu, Finland, and is a founding Editor-in-Chief of the International Journal on Document Analysis and Recognition.