{'fraction_non_alphanumeric': 0.039647876168277175, 'fraction_numerical': 0.009957229174681483, 'mean_word_length': 4.302375809935205, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Learning for Hybrid 5G Services in Mobile Edge Computing Systems: Learn from a Digital Twin

In this work, we consider a mobile edge computing system with both ultra-reliable and low-latency communications services and delay tolerant services. We aim to minimize the normalized energy consumption, defined as the energy consumption per bit, by optimizing user association, resource allocation, and offloading probabilities subject to the quality-of-service requirements. The user association is managed by the mobility management entity (MME), while resource allocation and offloading probabilities are determined by each access point (AP). We propose a deep learning (DL) architecture, where a digital twin of the real network environment is used to train the DL algorithm off-line at a central server. From the pre-trained deep neural network (DNN), the MME can obtain user association scheme in a real-time manner. Considering that real networks are not static, the digital twin monitors the variation of real networks and updates the DNN accordingly. For a given user association scheme, we propose an optimization algorithm to find the optimal resource allocation and offloading probabilities at each AP. Simulation results show that our method can achieve lower normalized energy consumption with less computation complexity compared with an existing method and approach to the performance of the global optimal solution.

. By achieving ultra-low end-to-end (E2E) delay and ultra-high reliability, URLLC lies the foundation for emerging latency-critical applications, such as factory automation, autonomous vehicles, and virtual/augmented reality [2]. Devices in these applications will generate some tasks that require to be processed within a short time. To reduce processing time at the local server of each device and to avoid delays in backhauls and core networks, mobile edge computing (MEC) is one promising solution [3]. However, when a task is packetized in a short packet and offloaded to a MEC server via a wireless link, the packet may be lost when the channel is in deep fading [4]. Besides, short blocklength channel codes will cause the none-zero decoding error rate, even for an arbitrarily high signal-to-noise ratio (SNR) [5]. Thus, achieving ultra-high reliability and ultra-low latency is very challenging in MEC systems.
On the other hand, mobile devices have only limited battery capacities. Improving the battery lifetime or energy efficiency (EE) of users is an urgent task [3,6,7]. By offloading tasks to MEC servers, we can save energy consumptions at the local servers (equipped at mobile devices), but extra energy is consumed for data transmissions. To minimize the total energy consumption of each user, we need to optimize the offloading probability. In a MEC network with multiple MEC servers and multiple users, the problem that optimizes user association, resource allocation, and offloading probabilities is non-convex and complicated. How to improve EE by solving a nonconvex problem in the scenario with both URLLC services and delay tolerant services remains an open problem.
To find the optimal solution to the problem, there are two kinds of approaches: optimization algorithms and machine learning algorithms. Since optimization algorithms need to search the optimal solution when channels change, they are suitable for small-scale problems, such as resource allocation in a single access point (AP) scenario [7,8]. When the scale of the problem grows, deep learning (DL) algorithms have the potential to find a near optimal solution in a realtime manner [9]. For example, a deep neural network (DNN) can be used as an approximator of an optimal resource allocation policy [9]. The resource allocation obtained from the optimal policy can be used as labeled samples to train the DNN. Once the training of the DNN is finished, we can compute the resource allocation from it with different channel realizations.
To train a DL algorithm, we first need to obtain optimal policies from simplified system models. However, optimal policies may not be available in practical systems. Thus, some other techniques are needed to enable DL algorithms. One approach that does not require labeled training samples is deep reinforcement learning [10]. By learning from the feedback of real- • We propose a digital twin enabled DL framework for improving EE of URLLC and delay tolerant services in a MEC system with multiple APs. The normalized energy consumption, defined as the energy consumption per bit, is minimized by optimizing user association, resource allocation, and offloading probabilities. With this framework, the optimal user association scheme is first explored in the DL framework, and then approximated by a DNN, which is first trained off-line at the central server and then sent the mobility management entity (MME). After the training phase, the central server keeps updating the DNN according to the variation of the real network environment. For a given user association scheme, each AP optimizes the resource allocation and task offloading policy. • We establish the digital twin of the MEC system, where the network topology, the channel and queueing models, and the fundamental rules are adopted to mirror the real system. In addition, the behavior of each AP, i.e., the optimal resource allocation and task offloading policy, is included in the digital twin. Although the problem at each AP is non-convex, we propose an algorithm that converges to the global optimal solution with linear complexity. • We design exploration policies that generate multiple user association schemes in each learning epoch. For each user association scheme, the normalized energy consumption, delay, and reliability are evaluated in the digital twin. Then, the best one is saved in the memory as labeled training samples. To compare the efficiency of different exploration policies, the normalized energy efficiency and computing complexity of different exploration policies are illustrated with simulation results.
Furthermore, simulation results show that the proposed DL framework can achieve a lower normalized energy consumption with less computing complexity compared with an existing solution, and can approach to the global optimal solution.

II. RELATED WORK
How to improve EE of mobile devices in MEC systems subject to the delay constraint has been widely studied in existing literature [7,8,[12][13][14]. To study the tradeoff between EE and latency, a weighted sum of energy consumption and latency was minimized in a single-AP scenario [7]. EE was maximized subject to the delay constraint in single-AP scenarios and multi-AP scenarios in [8] and [12], respectively. The authors of [13] analyzed the EE and latency with stochastic geometry and provided some useful guidelines to network provision and planning. The above studies mainly focused on one kind of services, and neglected the heterogeneities of services. To address this issue, a game theory approach was proposed in [14], where resource management and user association were optimized in multi-access MEC systems.
How to apply machine learning algorithms for user association or task offloading in MEC systems was also studied in some recent works [15][16][17][18]. Deep Q-learning was used to minimize the task execution cost by optimizing offloading decision according to channel state information, queue state information, and energy queue state of the energy harvesting system [15]. A similar method was also applied for energy harvesting of IoT devices in [16]. The authors of [17] proposed an efficient reinforcement learning-based resource management algorithm to incorporate renewable energy into MEC systems. More recently, a deep reinforcement learning framework for task offloading was studied in a single-AP scenario [18].
The above studies provided useful insights and promising machine learning algorithms in MEC systems, but they did not consider 5G services. Supporting URLLC in MEC systems was studied in [19] and [20]. In [19], the long-term average power consumption of mobile devices is minimized subject to the latency and reliability constraints. The weighted sum of delay and reliability is minimized in [20] for a single-user. Nevertheless, how to serve hybrid 5G services in MEC systems remains unclear, and deserves further study.

A. MEC System
We consider a MEC system shown in Fig. 1, where M APs serve K u URLLC services and K b delay tolerant services, which are indexed by K u = {1, ..., K u } and K b = {K u +1, ..., K u +K b }, respectively. For notational simplicity, we use a superscript ξ = {u, b} to represent the types of services in this work. If ξ = u, a parameter is used in URLLC services. Otherwise, it is used in delay tolerant services. All the notations used in this paper are listed in Table I. The APs are connected to the MME that is in charge of user association. To establish the digital twin, the MME sends some parameters and models of the network to a central server, where the user association scheme is optimized by a DL algorithm that learns from the digital twin. After the training phase, a DNN for user association is sent to the MME. With a given user association scheme, the network can be decomposed into single-AP problems. 1 For each 1 Frequency reuse factor is less than one that different bandwidth is allocated to adjacent APs. As such, there is no strong interference, and weak interferences are considered as noise.   single-AP problem, the AP optimizes resource allocation and task offloading for users that are associated with it.
Each AP is equipped with a MEC server and each user has a local server. Time is discretized into slots. The duration of each slot is T s . The service rates of the mth MEC server and the kth

B. Computation Tasks and Communication Packets
The computation tasks of the kth user are characterized by (λ ξ k , b ξ k , c ξ k ), where λ ξ k (packets/slot) is the average task arrival rate generated by the kth user, b ξ k (bits/packet) is the number of bits of each task (i.e., the size of a packet), and c ξ k (cycles/packet) is the number of CPU cycles required to process each task. We assume that each task is conveyed in one packet, and the relation between b ξ k and c ξ k is given by c ξ k = k 1 b ξ k , where k 1 > 0 (cycles/bit) depends on the computational complexity of the task [21][22][23].
For URLLC services, we assume that the packet size and the number of CPU cycles required to process each packet are constant (e.g., 32 bytes [1]), and the packet arrival process follows a Bernoulli process. In each slot, a user either has a packet to transmit or stays silent. For delay tolerant services, both the inter-arrival time between packets and the packet size may follow any general distributions. The only assumption is that the packet size of delay tolerant services is much longer than that of URLLC services. In the rest of the paper, the tasks of URLLC services and delay tolerant services are referred to as short and long packets, respectively.

C. Achievable Data Rate over Wireless Links
The users can offload tasks to one of the MEC servers via wireless links. Let β be the user association vector with entry β ξ m,k denoting whether the kth user is associated with the mth AP. If the kth user is associated with the mth AP, then β ξ m,k = 1. Otherwise, β ξ m,k = 0. We assume that each user can only offload packets to one of the APs, i.e., m∈M β ξ m,k = 1, where M = 1, ..., M is the set of indices of APs.

1) Achievable Rate for URLLC:
We consider orthogonal frequency division multiple access (OFDMA) systems. The number of subcarriers allocated to the kth user is denoted as N ξ m,k . Since the packet size of URLLC services is small, it is reasonable to assume that the bandwidth of N u m,k subcarriers is smaller than the coherence bandwidth and the transmission time is smaller than channel coherence time as well. Thus, each packet is transmitted over a flat fading quasistatic channel. If the kth user is accessed to the mth AP, the achievable rate of the kth URLLC user, k ∈ K u , can be approximated by [5] where W is the bandwidth of each subcarrier, α u m,k is the large-scale channel gain, g u m,k is the small-scale channel gain, P t,u k is the transmit power, Φ is a SNR loss coefficient, which reflects the gap between the achievable rate of practical channel codes and the approximation, N 0 is the single-side noise spectral density, f −1 Q is the inverse of Q-function, ε d,u k is the decoding error probability, and V u 2) Data Rate for Delay Tolerant Services: For delay tolerant services, the packet size is long, and Shannon's capacity is a good approximation of the achievable rate. If the kth user is accessed to the mth AP, the ergodic capacity of the kth user, k ∈ K b , can be expressed as where α b m,k is the large-scale channel gain, g b m,k is the small-scale channel gain, P t,b k is the transmit power.

1) Offloading Policy of URLLC Services:
Considering that feedback from receivers to transmitters may cause large overhead and extra delay, we assume that only 1 bit CSI is available at each transmitter, which indicates whether the small-scale channel gain is above a certain threshold, g th,u k . If the small-scale channel gain is above the threshold, then the packets are offloaded to the MEC with probability one. Otherwise, the offloading probability is zero. Thus, the overall offloading probability, x u k , equals the probability that g u m,k ≥ g th,u k , i.e., where Rayleigh fading is considered.

2) Offloading Policy of Delay Tolerant Services:
For each long packet, the transmission duration may exceed the channel coherence time. We consider an offloading policy that does not depend on the current small-scale channel gain. When the kth user, k ∈ K b , has a packet to process, the packet is offloaded to the MEC server with probability x b k ∈ [0, 1] and processed on the local server with probability (1 − x b k ).

E. Queueing Model
Local server  transmission. For URLLC services, each packet is transmitted in one slot. Since the packet arrival process follows a Bernoulli process, the peak arrival rate is one packet per slot, which is equal to the transmission rate of the wireless link. As a result, there is no queue before uplink transmission. For delay tolerant services, the peak arrival rate can be higher than the transmission rate, and hence some packets may wait in a communication queue before uplink transmission.
In the MEC servers, there are short and long packets. If the packets are served according to FCFS order, short packets arrive at the MEC servers after a long packet need to wait for the processing of the long packet. To avoid long queueing delay, a processor-sharing (PS) server is adopted at each AP [24]. On the PS server, the service rate of the server is equally allocated to all the packets in the server. When there are i packets in the mth server, the service rate of each packet is S m /i. As shown in [25], when there are short and long packets, the PS server outperforms the FCFS server.

F. Energy Consumption and Processing Rate at Local Server
Let e loc,ξ k be the energy consumption per CPU cycle of the kth user. According to the measurements in [21,26], e loc,ξ k = k 0 (C ξ k ) 2 (J/cycle), where k 0 is a coefficient depending on the chip architecture. The typical value of k 0 is 10 −15 . The energy consumption per packet at the local server is which indicates that the energy consumption for processing one packet increases with the processing rate C ξ k .

IV. PROBLEM FORMULATION AND DEEP REINFORCEMENT LEARNING FRAMEWORK
In this section, we first analyze the Quality-of-Service (QoS) constraints of two different services. Then, we formulate an optimization problem to minimize the maximum energy consumption per bit of all the users by optimizing user association, resource allocation, and offloading probabilities subject to the QoS requirements. Finally, we introduce the deep learning framework.

A. QoS Constraints of URLLC Service
The E2E delay of a packet is defined as the interval between the arrival time of a packet and the time when the processing of the packet is finished. For URLLC service, we denote D max,u and ǫ max,u as the required delay bound and the maximal threshold of the tolerable delay bound violation probability, respectively.

1) QoS Constraints on Local Servers:
If a packet is executed locally, the processing delay is When the channel is in deep fading, i.e., g u m,k < g th,u k , all the packets of a user is served by the local server and the arrival process is a Bernoulli process with average arrival rate λ u k . Given a constant service rate, C u k , the queueing model is a Geo/D/1/FCFS model. The complementary cumulative distribution function (CCDF) of queueing delay, D lq,u k , in Geo/D/1/FCFS model is given by [27] Pr{D lq,u The constraint on E2E delay can be expressed as follows, The queueing delay violation probability should satisfy which can be computed according to (6).

2) QoS Constraints When
Offloading to a MEC Server: When there are long and short packets in a PS server, an accurate approximation of the CCDF of the processing delay of short packets is given by [25], where ρ mc m is the workload of the mth MEC server, defined as follows, wherec b k is the average number of CPU cycles required to process a packet of delay tolerant services. The E2E delay of a packet when offloading to the MEC server should satisfy the following constraint, We set the upper bound of the decoding error probability and the upper bound of the processing delay violation probability to be equal, i.e., As shown in [4], setting different packet loss probabilities to be equal leads to minor power loss. By substituting processing delay violation probability in (9) into constraint ǫ mc,u k ≤ ǫ max,u 2 , we can derive the constraint on the workload as follows,

B. Stability of Delay Tolerant Services
For delay tolerant services, we only need to ensure the queueing system is stable, i.e., the average service rate is equal to or higher than the average arrival rate.

1) Rate Constraint of Local Servers:
To ensure the stability of the queueing system on local servers, we need to guarantee that the processing rate is higher than the average data arrival rate, Besides, the processing rate should not exceed the maximal computing capacity of the server, 2) Rate Constraint of Wireless Link: To ensure the stability of the communication queue in Fig. 2, we need to guarantee that the average transmission rate of the wireless link is equal to or higher than the average data arrival rate, i.e., whereb b k is the average number of bits in a long packet.

3) Workload Constraint on the MEC Server:
In the case that only delay tolerant services offload packets to the mth MEC server, x u k = 0, ∀k ∈ K u , the stability of the PS server can be satisfied if the workload meets the following constraint, Otherwise, constraint (13) should be satisfied.

C. Objective Function: Normalized Energy Consumption
Our goal is to minimize the normalized energy consumption, defined as the energy consumption per bit.

1) URLLC Services:
For URLLC services, the circuit power at the local server and the average transmit power for packets offloading are λ u k E loc,u k and λ u k P t,u k T s (J/slot), respectively. Since the average data arrival rate is λ u k b u k (bits/slot), the normalized energy consumption is 2) Delay Tolerant Services: If a packet is processed at the local server, the average energy k , which is obtained from (4). Then, the energy consumption per bit is η loc,b If the packet is offloaded to a MEC server, the energy consumption and the average amount of data transmitted in each slot can be expressed as P t,b k T s and k , respectively. Then, the energy consumption per bit is η mec,b k . Therefore, the normalized energy consumption of user k, k ∈ K b , can be expressed as follows,

D. Optimization Problem
To avoid the users with bad channel conditions or high task arrival rates experiencing high energy consumption, we take fairness among all the users into consideration by minimizing the maximal normalized energy consumption of the K u + K b users. If there is a central control plane that manages user association and resource allocation, the optimization problem can be formulated as follows, (1), (2), (7), (8), (11), (12), (14) and (15), where N max is the total number of subcarriers of each AP and P max,ξ k is the maximal transmit power of the kth user. Constraint (19d) is obtained from (13) and (16). Since the required transmit power is determined by the bandwidth allocation and the offloading probability, it can be removed from the optimization variables. The relation between the optimal solution and the inputs, i.e., large-scale channel gains and average task arrival rates, is denoted as and (·) T denotes the transpose operator. In practice, a user can subscribe to both kinds of services. If the kth user subscribes to both kinds of services, λ u k and λ b k are referred to as the average task arrival rates of URLLC and delay tolerant services, respectively. The large-scale channel gains of the two kinds of services are the same, i.e., α u k = α b k . In the local server of the kth user, the packets from different services are waiting in two separated FCFS queues. The energy consumption per packet in (4) becomes The transmit power constraint of the kth user becomes P u The rest of the constraints remains the same.
Note that user association is managed by MME, but resource allocation and offloading probabilities are determined by each AP. The problem P 1 is decomposed into two subproblems that are solved in two timescales at MME and APs, respectively. In the first subproblem, each AP optimizes resource allocation and offloading probabilities with given user association scheme.
In the second subproblem, the MME optimizes user association scheme with a DL algorithm, where the behavior of each AP (i.e., the optimal resource allocation and task offloading policy) is taken into account.
• Problem P 2 : The problem that optimizes subcarrier allocation and offloading probability can be formulated as follows,  (14) and (15).

E. Structure of Deep Learning
It is worth noting that both P 2 and P 3 are non-convex. We will propose an optimization algorithm to solve problem P 2 and apply the deep learning algorithm to solve problem P 3 . The framework of the digital twin enabled DL algorithm is illustrated in Fig. 3. The inputs of the DNN are the large-scale channel gains from users to APs and the average task arrival rates of all the users, while the output of the DNN is the user association scheme. The direct output of the DNN is denoted asβ, based on which we explore user association schemes. For a user association scheme, we can obtain the related normalized energy consumption from the digital twin. From the feedback of the digital twin, we find the best user association scheme,β, that minimizes the normalized energy consumption among the user association schemes randomly generated according to exploration policies. Finally, the inputs α, λ and the best outputβ are saved in the memory and will be used to train the DNN.

V. ALGORITHM TO SOLVE PROBLEM P 2
In this section, we propose a method to find the optimal solution of problem P 2 . Note that when the user association scheme is given, problem P 2 can be decomposed into multiple single-AP problems. In this section, we omit index m for notational simplicity.  Table II.
If the required bandwidth is larger than N max , η th cannot be achieved, and the minimum of max k∈K ξ η ξ k is higher than η th . Otherwise, the minimum of max k∈K ξ η ξ k is lower than η th . Via binary search, η th converges to the minimum of max k∈K ξ η ξ k , and the corresponding bandwidth allocation and offloading probabilities are the optimal solution of problem P 2 (See proof in Subsection V-C.).
In the second step, we remove constraint (19d), and decompose problem (22) into multiple single-user problems. For each single-user problem, we search the minimum number of subcarriers allocated to each user via binary search. For a given value of N th,ξ k , we minimize η ξ k subject to N ξ k = N th,ξ k , i.e.,  Require: Large-scale channel gains, α, the user association scheme, β, and the required searching precisions of normalized energy consumption, number of subcarriers, and offloading probability, ση, σN , and σx.     Table II.
If constraints (19b) and (19d) cannot be satisfied by the end of the binary search, problem P 2 is infeasible and the AP cannot guarantee the QoS requirements of all the users associated with it. In this case, the normalized energy consumption of the user association scheme will be set as infinite in the learning framework in Fig. 3. This user association scheme will not be used to train the DNN since the QoS requirements cannot be satisfied.

B. Optimal Offloading Probability
In this subsection, we show how to solve problem (23). Since the offloading probability depends on bandwidth allocation, we denote the optimal offloading probability asx ξ k (N th,ξ k ). 1) URLLC Services: For URLLC services, the offloading probability is determined by the threshold of small-scale channel gain g th,u k . To find the optimal offloading probability, we optimize g th,u k by the following three steps to meet all the constraints in problem (23).
In the first step, we find the minimal energy consumption per packet at the local server. Since the normalized energy consumption increases with the service rate, we first find the minimal service rate that is required to satisfy the constraints on the E2E delay and the queueing delay violation probability, i.e., (7) and (8) (6), the minimal service rate can be obtained when ǫ lq,u k = ǫ max,u . We denote the minimal service rate that is required to satisfy D lq,u k and ǫ lq,u k as C u * k . According to (4), the minimal energy consumption per packet at the local servers is E loc,u * k = k 0 (C u * k ) 2 c u k (J/packet). In the second step, we find the minimal value of g th,u k that can satisfy the constraints on decoding error probability and maximal transmit power, i.e., ǫ d,u k ≤ ǫ max,u 2 in (12) and (19f). The decoding error probability can be obtained from (1) by setting T s R u k = b u k . Then, the required transmit power that satisfies ǫ d,u k = 1 2 ǫ max,u can be expressed as follows, where and the approximation V u k ≈ 1 is applied, which is accurate when the receive SNR is higher than 5 dB [28,29]. To satisfy the maximal transmit power constraint, we can obtain the minimal g th,u k by substituting (24) into P t,u k = P max,u , i.e., In the third step, we derive the closed-form expression of the optimal threshold,ĝ th,u k , that minimizes the normalized energy consumption. Substituting x u k = e −g th,u k and P t,u k = 1 (17), we can derive the derivative of η u k on x u k as follows, From (27), we can see that the sign of η u ′ k is the same as k . Therefore, η u k first strictly decreases and then strictly increases with g th,u k , and there is a unique solution of g th,u k that minimizes η u k (i.e., f (g th,u k ) = 0). The solution of f (g th,u k ) = 0 can be derived as follows, If g min,u k ≤g th,u k , theng th,u k is the optimal threshold that minimizes η u k subject to the transmit power constraint. Otherwise, since η u k increases with g th,u k in the region (g th,u k , ∞), g min,u k is the optimal threshold. Thus, we haveĝ By substitutingĝ th,u k into (3), we can obtain the optimal offloading probability,

2) Delay Tolerant Services:
We apply the binary search to find the optimal offloading probabilities of delay tolerant services that meet the constraints of problem (23). Given N th,b k , the upper bound of the offloading probability that satisfies the constraints on average data rate and maximal transmit power in (15) and (19f) can be obtained by substituting P t,b The lower bound of the offloading probability that satisfies the service rate constraint at the local server in (14) can be obtained by substituting , the problem is infeasible, which may happen when the average packet arrival rate λ b k is large. When the problem is feasible, to find the optimal offloading probability, , we need the following proposition, Proof. See proof in Appendix A.
Then, the optimal offloading probability,x b k (N th,b k ), that minimizes η b k can be obtained via binary search.

C. Convergence of the Algorithm
In this subsection, we first prove that for a given threshold of the normalized energy consumption, η th , the algorithm in Table II can find the minimum bandwidth that is required to achieve the threshold (From Line 4 to Line 22 in Table II). To prove it, we only need to prove that the normalized energy consumption decreases with N th,ξ k .
Property 1. The minimum of the objective function (23) decreases with N th,ξ k in the region Proof. See proof in Appendix B.
The above property indicates that the binary search converges to the minimal N th,ξ k that can guarantee η k ≤ η th , unless it is infeasible (as shown in Line 20 of Table II).
To find out whether problem P 2 is feasible or not, we minimize the total number of subcarriers and see whether it is less than the total number of subcarriers of an AP. Besides, we also need to minimize the total offloading probability and see whether it satisfies the constraint in (19d). 2 The following property indicates that minimizing the offloading probability of the kth user is equivalent to minimizing the number of subcarriers allocated to it.
Proof. See proof in Appendix C.
Therefore, by minimizing the sum of the numbers of subcarriers, we also obtained the minimum of the sum of the offloading probabilities. In other words, both the sum of the numbers of subcarriers and the workload at the MEC server are minimized with the algorithm from Line 4 to Line 22 in Table II. As a result, problem P 2 is feasible if and only if constraints (19b) and (19d) are satisfied with N th,ξ k andx ξ k (N th,ξ k ), i.e., the condition in Line 23 in Table II. If problem P 2 is feasible when the normalized energy consumption equals η th , then η th is achievable and the minimal normalized energy consumption η * ≤ η th . Otherwise, η * > η th . Therefore, with the binary search (i.e., Lines 2,3 and Lines 23 to 27), η th converges to η * . The coresponding N th,ξ k andx ξ k (N th,ξ k ) converge to the optimal solution N ξ * k and x ξ * k .

D. Complexity Analysis
Given the required searching precision of the normalized energy consumption σ η , it takes O log 2 ( η ub ση ) steps to obtain the minimum of the maximal normalized energy consumption of all the users. To achieve a target η th , (K u + K b )O log 2 ( N ub,ξ k σ N ) steps are needed to obtain the required numbers of subcarriers of K u + K b users, where σ N is the required searching precision of the number of subcarriers. For a given number of subcarriers, it takes O log 2 ( 1 σx ) steps to obtain the optimal offloading probability that minimizes the normalized energy consumption of the delay tolerant user in the region [0, 1], where σ x is the required searching precision of offloading probability. For URLLC services, the optimal offloading probability can be obtained in the closed-form expression in (29). Therefore, the complexity of the algorithm can be expressed , which increases linearly with (K u + K b ).

VI. DEEP LEARNING FOR USER ASSOCIATION
In this section, we discuss how to explore user association schemes and how to train the DNN.
The set of all the weights and biases of the DNN is denoted as where L dnn is the number of layers, W [l] and b [l] are the weights and the biases in the lth layer, respectively. The relation between the input and output of the lth layer can be expressed as where X [l] and Y [l] are the input and output of the lth layer, and the activation function, f δ (x), is an element-wise operation of a vector. In this work, we use ReLU function as the activation function, i.e., f δ (x) = max(0, x).
In each learning epoch, the large-scale channel gains, α, and the average task arrival rates, λ, are estimated by the system, and are used to calculateβ from the DNN with parameters Θ. With July 3, 2019 DRAFT the outputβ, user association schemes are generated according to the exploration policies. Then, we find the best user association scheme that minimizes the normalized energy consumption.
The pair of inputs α, λ and the best user association scheme, denoted asβ, are saved in the memory, and will be used to train the DNN. By the end of the epoch, N t training samples, (α, λ,β), are randomly selected from the memory to train the DNN. After the training, Θ is updated for the next epoch.

A. Exploitation and Exploration of the DNN
With ReLU function, the outputs of the DNN are continuous variables, i.e.,β ξ k = (β ξ 1,k , ...,β ξ M,k ) T . We first discuss how to explore user association schemes based on the outputs, and validate the impacts of exploration policies on the normalized energy consumption with simulation.

1) Highest Value (Exploitation):
For the kth user, a direct way to map the continuous variableŝ β ξ k to discrete a user association scheme is to access to the AP with the highest output. We denote the index of the AP with the highest output as m * k = arg max m∈Mβ ξ m,k . Then, β ξ m * k ,k (0) = 1 and β ξ m,k (0) = 0, ∀m = m * k . The user association scheme is denoted as β(0).

2) One
Step Exploration: Based on β(0), we change the association scheme of one of the K u + K b users, while the association scheme of the other users remains the same as β(0).
Since only one user changes the scheme, this method is referred to as one step exploration.

3) Random Exploration:
With the random exploration policy, each user randomly selects one of M APs with probability 1/M. The user association schemes generated with this method are denoted as β(µ OS + 1), ..., β(µ OS + µ RE ), where µ RE is the number of schemes generated with the method.

B. The DNN Training
From the 1 + µ OS + µ RE user association schemes, we choose the one that minimizes the normalized energy consumption,β = arg min i=0,1,...,µ OS +µ RE Q * 2 (α, λ, β(i)|π 2 ), and save it in the memory. The memory is empty at the beginning of the first epoch, and the initial values of parameters in Θ follow a zero-mean normal distribution. When the memory is full, the newly obtained training set, (α, λ,β), replaces the oldest one.
We adopt the experience replay technique in [30] to train the DNN using N t training samples.
The parameters in Θ are updated by using the Adam algorithm [31] to reduce a training loss function, defined as L(Θ) = − 1 Nt Nt nt=1 (β nt ) T log(β nt ) + (1 −β nt ) T log(1 −β nt ) . When the value of L(Θ) is below a required threshold, σ L , the training phase is finished. After the training phase, MME can use the DNN to calculate user association scheme for any α, λ.

A. Simulation Setup
The real network topology that will be used in our simulation is illustrated in Fig. 4. We vary the user distribution ratio, defined as the user density in region 1 to the user density in region 2, to see how the impacts of user distribution on normalized energy consumption. The path loss model is 35.3 + 37.6 log 10 (d), where d is the distance (meters) between an AP and a user [32].
The shadowing is lognormal distributed with 8 dB standard deviation. The small-scale channel fading follows Rayleigh fading. The packet arrival rate of delay tolerant users, λ b k , is uniformly distributed between 5 and 10 packets/s. The packet arrival rate of URLLC users, λ u k , is 500 packets/s [33]. Simulation parameters are summarized in Table III, unless mentioned otherwise.

B. Optimal Bandwidth Allocation and Offloading Probabilities
In this subsection, we show the normalized energy consumption achieved by the optimal bandwidth allocation and offloading probability with given user association scheme. In this case, we only need to consider single-AP scenarios. The users are randomly distributed around the AP.
Since there is no existing method that optimizes bandwidth allocation and offloading probability for both URLLC and delay tolerant services, we compare the proposed method (with legend 'Proposed') with two baselines. In the first baseline, the bandwidth allocation is the same as the optimal solution, but all the packets are offloaded to the MEC (with legend 'MEC'). In the second baseline, all the packets are processed at the local servers (with legend 'Local'). The normalized energy consumption depends on the location of users and shadowing. In this subsection, we generate 200 α, λ randomly and calculate the average normalized energy consumption.

C. DL Algorithm for User Association
In this subsection, we show the normalized energy consumptions of different user association schemes, where N max = 48, S = 0.4 GHz, and K b = K u = 5. We compared our DL algorithm (with legend 'DL') with the optimal user association scheme (with legend 'Optimal') that is obtained by exhaustively searching for all possible user association schemes. To show the impacts of non-stationary environment on the performance of the DL algorithm, we also provide the performance of a well-trained DNN that will not be updated when the user density varies (with legend 'DL Fixed DNN'). With this scheme, there is no exploration and the output of the DNN will be used as the user association scheme. Some similar studies focused on offloading and resource allocation with a single AP [8,18]. The implicit assumption on the user association is that the users are served by the nearest AP or the AP with the highest large-scale channel gain. In addition, a game theory approach was proposed to optimize resource management and user association in [14]. Thus, we compared the proposed method with three baselines: With the first baseline method, users are served by the nearest AP (with legend 'Nearest AP'). With the second baseline method, users are connected to the AP with the highest large-scale channel gain (with legend 'Highest α'). With the third baseline, the game theory approach based on a coalition game in [14] is used to iteratively optimize user association (with legend 'Game').
Following [14], we set the number of the coalitions as M (the number of APs) and the users are randomly chosen to perform Merge, Split and Exchange operations by preferring a smaller objective function as in (21).
We provide the simulation results in scenarios with different numbers of APs: M = 2 and M = 3. When M = 2, the optimal scheme can be obtained with the exhaustive searching method. However, when M = 3, the complexity of the exhaustive searching method is too high, and we cannot obtain the optimal scheme. are shown in the legends, e.g., 'DL{10, 100}' means µ OS = 10, µ RE = 100. The results in Fig.   7 show that our proposed method can achieve much smaller normalized energy consumption than the three baseline methods, and perform close to the optimal scheme. For the 'Game' scheme, it converges after 100 iterations in average. As indicated in [14], this scheme needs to evaluate the objective function twice in each iteration, which means it needs to evaluate the objective function around 200 times. However, our proposed algorithm 'DL{10, 100}' only needs to explore (10 + 100) user association schemes, i.e., evaluating the objective function 110 times, which is less than the 'Game' scheme. Therefore, our proposed algorithm can achieve a lower normalized energy consumption with less computation complexity. We can also observe that directly exploit the output of DNN without any exploration can save around 30 % normalized energy consumption compared with 'Highest α'. With a few more explorations, the performance can be further improved as shown by 'DL{10, 100}'. Moreover, the one step exploration policy 'DL{10, 0}' can achieve lower normalized energy consumption with less explorations compared with the random exploration policy 'DL{0, 100}'. These results indicate that the output of the DNN can help improving the efficiency of the exploration policy.
In Fig. 8, we study the impacts of the variation of the user density on the proposed DL algorithm. In the digital twin, the user distribution ratio is set to be 5 : 5. After 1000 tests with 5 : 5 user distribution, the user distribution ratio in the real network becomes different, i.e., 9 : 1.
The MME needs to update DNN according to the variation of the user distribution. The legends of DL algorithms with the user distribution variation are followed by (5 : 5  The normalized energy consumptions achieved with different schemes in the scenario with 3 APs are provided in Table IV. We compare the average of the normalized energy consumptions in the last 1000 epochs with the 'Nearest AP', the 'Highest α' and the 'DL' schemes. The results in Table. IV show that the 'DL' scheme can save around 72 % and 59 % normalized energy consumption compared with the 'Nearest AP' and the 'Highest α' schemes, respectively. This observation indicates that our proposed framework can find an efficient user association scheme when M = 3.

VIII. CONCLUSION
In this work, we studied how to reduce the normalized energy consumption of users with URLLC and delay tolerant services in a MEC system. We proposed a DL architecture for user association, where a digital twin of network environment was established at the central server for training the algorithm off-line. After the training phase, the DNN was sent to the MME that manages user association. With a given user association scheme, we proposed a low-complexity optimization algorithm that optimized resource allocation and offloading probabilities at each AP.
Simulation results indicated that by optimizing resource allocation and offloading probability, our low-complexity algorithm can save more than 87 % energy compared with the baselines.
Besides, with the DL algorithm, our user association scheme can achieve lower normalized energy consumption with less computing complexity compared with an existing method and approach to the global optimal solution. APPENDIX A

PROOF OF PROPOSITION 1
Proof. Substituting the equality in (14) into (4), we can obtain that E loc,b k = k 0 (λ b k ) 2 (c b k ) 3 . Further substituting E loc,b k into the first term in (18), we can derive the normalized energy consumption at the local server, i.e., From (A.1), we can derive that k ) > 0, and hence the first term in (18) is convex in x b k . To prove the second term in (18) is convex in x b k , we only need to prove P t,b k is convex in (15), the required average data rate linearly increases with x b k . Thus, we only need to prove P t,b k is convex in E R b k . From (2) we can see that R b k increases with P t,b k , and it is concave in P t,b k . Since the expectation does not change the monotonicity and convexity of the function, E R b k is an increasing and concave function with respect to P t,b k . According to [34], if the original function is an increasing and concave function, then the inverse function is a convex function. Therefore, P t,b k is convex in E R b k . Since the two terms in (18) The proof follows.
APPENDIX B

PROOF OF PROPERTY 1
Proof. For URLLC services, according to [29], we know that the required transmit power P t,u k to achieve a certain service rate decreases with N th,u offloading probability x u k = 0 and N th,u k ≤Ñ u k . For delay tolerant services, we need to guarantee the rate constraint of the wireless link in (15).
k λ b k /T s , and obtain the relationship between E g b k R b k and N b k when the offloading probability x b k = 0 as follows That is for a given average offloading packet rate x b k λ b k , the transmitting power P t,b  (18) is applied. (C.1) indicates that to prove η b ′ k decreases with N b k , we only need to prove that (2) into the rate constraint of the the wireless link in (15), we can derive that Based on (C.2), we can see that k is an increasing function of N b k . According to the characteristic of inverse function (i.e.,