{'fraction_non_alphanumeric': 0.04325803684490013, 'fraction_numerical': 0.008251296924145255, 'mean_word_length': 4.359457092819615, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
QoS-Aware Scheduling in New Radio Using Deep Reinforcement Learning

Fifth-generation (5G) New Radio (NR) cellular networks support a wide range of new services, many of which require an application-specific quality of service (QoS), e.g. in terms of a guaranteed minimum bit-rate or a maximum tolerable delay. Therefore, scheduling multiple parallel data flows, each serving a unique application instance, is bound to become an even more challenging task compared to the previous generations. Leveraging recent advances in deep reinforcement learning, in this paper, we propose a QoS-Aware Deep Reinforcement learning Agent (QADRA) scheduler for NR networks. In contrast to state-of-the-art scheduling heuristics, the QADRA scheduler explicitly optimizes for the QoS satisfaction rate while simultaneously maximizing the network performance. Moreover, we train our algorithm end-to-end on these objectives. We evaluate QADRA in a full scale, near-product, system level NR simulator and demonstrate a significant boost in network performance. In our particular evaluation scenario, the QADRA scheduler improves network throughput by 30% while simultaneously maintaining the QoS satisfaction rate of VoIP users served by the network, compared to state-of-the-art baselines.


I. INTRODUCTION
Traffic in cellular networks has increased dramatically in the recent decades. The fifth generation (5G) mobile network, also known as New Radio (NR), is designed to further increase traffic capacity, provide support for new use-cases, and enhance development of a diverse set of applications, including internet of things (IoT) applications and autonomous and/or remotely controlled systems [1]. With an increasingly diverse set of applications in NR networks, maintaining the quality of service (QoS), compared to previous generations of cellular networks, is bound to become an even more challenging task. The scheduler is one of the core components of cellular networks and controls the allocation of the finite set of network resources, i.e. time, frequency, and spatial resources, to user equipment (UE) in both uplink and downlink transmissions. The development of efficient schedulers is therefore key for efficient usage of network resources and to overall network performance [2].
The scheduling algorithm is typically separated into two independent operational components, a time-domain (TD) unit, selecting the data flows to be served in each scheduling instance, and a frequency-domain (FD) unit, which allocates communication resources in the physical radio domain to the selected data flows [2]- [4]. This is shown in Fig. 1 and will be further detailed in Sec. II. Current state-of-the-art TD scheduling algorithms are based on heuristics that assign numerical weights to data flows based on properties such as channel quality, head-of-line delay and average throughput. A data flow is then given priority based on its weight, relative to other data flows [2]- [4]. Data flows are thus sorted, implicitly, based on the assigned weights. However, sorting data flows based on heuristics defined through individual data flows may ultimately result in a sub-optimal scheduling strategy. Firstly, by failing to account for the composition of data flows, this approach is unable to consider the constraints enforced on the FD unit, i.e. physical resource constraints.
Secondly, heuristic approaches are unable to explicitly optimize for network performance indicators, e.g. QoS satisfaction rate and network throughput. Moreover, with the increased traffic diversity in NR networks as well as the increased flexibility offered by the NR specification [1], reliably maintaining and tuning network performance through heuristics might become a complex and daunting task.
To overcome these issues, in this paper, we propose a new approach to TD scheduling for NR using recent advancements in deep reinforcement learning (DRL) [5]. Using DRL, we formulate, train, and evaluate a TD scheduling algorithm that 1) explicitly sorts data flows by taking all data flows into account as well as their QoS requirements, 2) explicitly optimizes the aggregate network performance as well as QoS satisfaction rate, 3) is trained end-to-end on the optimization objective, and 4) allows for fine-granular control of fundamental network performance trade-offs by configuring the training objective with a, so-called, preference vector. We denote our approach as QoS-Aware Deep Reinforcement learning Agent (QADRA) for NR Time-domain Scheduling.
The remainder of this article is structured as follows. First, in Section II, we discuss the scheduling model used in which we develop our algorithm. In Section III, we provide a brief overview of DRL and in Section IV we present our main contribution, the QADRA scheduler. Finally, we evaluate the approach in Section V using a full-scale NR system level simulator, and conclude the paper in Section VI.

II. MODEL AND OBJECTIVES
Consider a TD-FD scheduler tasked with assigning physical radio resources to data flows [2]- [4]. We denote a data flow with f := {x, q}, where x ∈ R denotes the data flow feature vector in some feature space R and q ∈ Q denotes the flow QoS requirements in some space of QoS requirements, Q. Data flows that lack strict QoS requirement, such as file transfers, are referred to as best effort traffic. At every transmission time interval (TTI) indexed by t = 1, 2, . . . , the scheduling process is executed in the following manner: the scheduler first receives an input set of N t data flow feature vectors X t := {x t,j } Nt j=1 . The TD scheduler compiles an output list Y t of data flows by sorting the elements of X t in a decreasing order of priority inferred from their respective feature vectors. Next, the FD scheduler parses Y t serially and assigns physical radio resources to each data flow until the available resources are exhausted. The scheduled data flows are then transmitted over their corresponding physical resource assignments. This is summarized in Fig. 1.
Physical radio resources carry the control channel as well as the data channel. The control channel contains the information required to locate and decode the data payload, carried by the data channel, at the receiver. In state-of-the-art cellular networks, the physical resources are divided between the physical uplink and downlink control channels (PUCCH and PDCCH, respectively), and the physical uplink and downlink shared channels (PUSCH and PDSCH, respectively). The typical FD scheduling process hence consists of two sequential steps: control resource allocation and data resource allocation, as seen in Fig. 1. With control resource allocation, the FD scheduler assigns PDCCH and PUCCH resources to the highest prioritized data flows, as defined by the TD scheduler. The FD scheduler then tries to allocate data channel resources, PUSCH or PDSCH, to these data flows. Starting with the most prioritized data flow, data channel resources are assigned if available, otherwise, its assigned control channel resources are deleted. The process is then repeated for each data flow until all data flows with control channel allocation have been attempted to be given data channel resources.
The delay experience by a data packet, which is served by a data flow, is defined as the time between the creation and the successful reception of the packet by the intended receiver. Further, the average network throughput is defined in terms of the number of bits successfully transmitted over the network per time unit.

III. DEEP REINFORCEMENT LEARNING
Reinforcement learning (RL) [6] is a paradigm of artificial intelligence that deals with learning optimal (sequential) decision making in dynamic environments. It is based on Markov Decision Processes (MDPs), characterized by a state space S, action space A, state transition function T (s , s, a) = P S t+1 = s S t = s, A t = a , reward function R(s , s, a), and discount factor γ ∈ [0, 1). The general RL algorithm aims to learn a policy distribution that maximizes the (expected and discounted) accumulated reward: where A t ∼ π(S t , ·) and S t+1 ∼ T (·, S t , A t ). Note, the transition function, T , and reward function, R, are unknown a priori, thus requiring an algorithm to explore the environment dynamics in order to learn the optimal action for an arbitrary state.

A. Deep Q-Networks
Deep Q-Networks (DQNs) [5] were one of the first approaches to incorporate deep neural networks into an RL algorithm. DQNs are based on Q-learning [7], and use deep neural networks to learn the value of executing a certain action, a, in a given state, s, Q(s, a). In particular, the Q-function, Q : S × A → R, denotes the (expected and discounted) accumulated reward following a specific state-action pair under an optimal policy, with S t+1 ∼ T (·, S t , A t ) and A t ∼ π * (S t , ·). Given the Qfunction, an optimal policy is given by where a * = arg max a∈A Q(s, a) and where 1{·} denotes the indicator function.

IV. TIME DOMAIN SCHEDULING USING DEEP REINFORCEMENT LEARNING
In this section we present the QADRA scheduling algorithm, a new approach to TD scheduling in NR using DRL. At a high level, QADRA is a machine learning-guided list sorting algorithm, which is trained using the feedback loop shown in Fig. 2.
Given the scheduling process discussed in Sec. II, the TD scheduling problem reduces to sorting of data flows. Current scheduling algorithms approach this problem by assigning weights to each data flow using heuristics [2]- [4], and subsequently sort the data flows, implicitly, based on their assigned weights. We wish to construct an algorithm that learns to explicitly sort data flows such that the final resource allocation is optimal in terms of the user QoS satisfaction rate.
We first define, through the reward function, the optimization objective of our algorithm. Avoiding any form of reward shaping, we define the reward based on the fulfillment of flow QoS requirements, encoded by q t,j , for flow j during TTI t as Best effort traffic, on the other hand, is given a reward equal to the number of bits transferred successfully. Next, we form a reward vector . . , N G } denotes a data flow type (i.e., a group of flows with identical QoS requirements, e.g. conversational video), N G denotes the total number of data flow types, and the total reward gathered by all flows of type G i during TTI t. Finally, in order to reduce the vectorized reward, r t , into a scalar reward, r t , we introduce a preference vector, ω ∈ R N G , and let r t = ω T r t .
The preference vector, thus, provides a way to explicitly control the trade-off between conflicting QoS requirements by modifying the optimization objective. For example, letting A. List sorting using deep reinforcement learning A key difficulty in applying RL to list sorting is the size of the action space. Given a set of N unique elements, x 1 , . . . , x N , there are N ! possible unique sequences, hence an action space of size N !. We circumvent this problem by reducing the sorting problem from one action, in a space of size N !, into a sequence of actions in much smaller spaces. In particular, the selection sort algorithm, shown in Algorithm 1, sorts a list of N unique elements in N steps. At each step k = 0, 1, 2, . . . , N − 1, the input list, X , contains N − k elements from which a policy selects one to append to the output list, Y. Thus, using selection sort, N actions in action spaces of sizes N, N − 1, . . . , 1 are sufficient.
The selection of a k ∈ {0, 1, . . . , N − k} (line 3 of Algorithm 1) fully determines the output order. For example, when sorting numerical lists, x j ∈ R ∀j, letting a k = arg min j x j results in an output list in ascending order. In similar fashion, N t data flows can be sorted using a policy trained, e.g., using DRL. The policy is given the current input and output lists (of flow feature vectors) and returns the index of the data flow of the input list next to append to the output list. In other words, the state space is given by S = S i × S o , where S i and S o denote the state spaces of the input and output lists, respectively. Since the two lists both contain data flows, S i = S o . Each data flow is characterized by a feature vector x ∈ R and a list may contain any number of data flows, hence where 0 denotes an empty list. The action space depends on the current input list. Given N t data flows to sort, the action space of the kth action is given by A t,k = {0, 1, . . . , N t − k}.

B. The QADRA Scheduler
In this section, we describe the QADRA scheduler, which uses a selection sort approach to optimally schedule data flows in every TTI, in terms of the reward defined previously in this section. QADRA implements a, slightly modified, DQN as its selection policy. The states of the input and output lists are reduced to a state vector, s, by sequentially passing each element of the respective lists through a recurrent neural network. A state-action pair is then evaluated by passing the state vector and the feature vector of the corresponding action through a feed-forward network, from here on denoted the Q-Network, as illustrated in Fig. 3. Moreover, since sequential information is required to form the correct states during training, sequences of transitions are stored in the replay buffer used by the DQN algorithm, as opposed to individual transitions.
The QADRA scheduler consists of two distinct processes, allowing for distributed training, an actor process and a learner process. These are detailed below. During evaluation, only the actor process is run. We denote the recurrent networks using where d represents the internal state of the network, x the input feature vector, and s denotes the output from the network. Two sets of parameters, φ, θ, are used, one for the input and one for the output list. The Q-Network is parameterized by ψ. When referring to all sets of parameters, we use Γ = {φ, θ, ψ}.

C. Actor process
The actor process is outlined in Algorithm 2. The actor explores the environment and stores sequences of states, actions and rewards into a replay buffer, then used by the learner process. The algorithm is given (and occasionally updated throughout training) sets of parameters φ, θ, ψ as well as an exploration rate ∈ [0, 1]. During each TTI t, N t data flows are to be sorted whose feature vectors {x t,k } Nt k=1 form the input list X t . In order to save the initial set of data flows to the replay buffer, the input list is first cloned into X − t . An output list, Y t , is initialized as well as internal states of the recurrent networks, d i , d o , and state vectors, s i , s o , with sub-scripts i and o referring to input and output lists, respectively.
The input state vector, s i , is first calculated by sequentially passing each feature vector, x t,k , through the recurrent network parameterized by φ. The input list is then sorted using N t actions, chosen using an -greedy policy, as outlined in Section IV-A. Following an action, the output state vector, s o , is updated accordingly. After N t actions, the output Y t is given to the FD unit. A reward, r t , is computed using the pre-defined preference vector, as previously discussed, and a new set of data flows, X t+1 , are obtained. Finally, the observed sequence (X − t , a t,1 , . . . , a t,Nt , r t , X t+1 ) is added to the replay buffer.

D. Learner process
The learner process is outlined in Algorithm 3. The learner samples experiences, produced by the actor, from a replay buffer and updates the parameters of the networks to more accurately predict the value of state-action pairs, Q(s, a). The algorithm is given a recurrent network, h, with two sets of parameters, φ, θ, and a Q-Network with corresponding parameters ψ. These networks are fixated (and occasionally refixated throughout training) with parameters φ − , θ − , ψ − and referred to as target networks (cf. [5]). Each training step, an experience (X τ , a τ,1 , . . . , a τ,Nτ , r τ , X τ +1 ) is sampled from the replay buffer. The input and output states, s i , s o , and their target network equivalences, s − i , s − o , are initialized as well as the internal states of the recurrent networks, Algorithm 2: Actor process Data: Recurrent network h with two sets of params φ, θ. Feed-forward network Q with params ψ. Exploration rate . 1 Let s denote the concatenation of s i and s o ; 2 Acquire a set of data flows Initialize empty output list Y t ; 6 Reset states: Add sequence (X − t , a t,1 , . . . , a t,Nt , r t , X t+1 ) to replay buffer;

18
Occasionally: update φ, θ, ψ from the learner process; 19 end First, the input states, s i , s − i are computed by passing each data flow of X τ through the recurrent network, using parameters φ, φ − , respectively. Then, at each step k, the target value must be computed, hence the target output state is first updated using the action a τ,k , followed by the accumulation of gradient, ∇ Γ, on the squared error between the value of the chosen stateaction pair and the value of the following state. Then, the output state is updated and prepared for the next action. The last action in each sequence is treated differently. A non-zero reward is given, hence included in the loss function and the next state is given by the next set of data flows to sort, X τ +1 . Hence, the target input state must be recomputed, and the target output state reset. Following this step, the accumulated gradient is normalized by the sequence length, N τ , and a gradient descent step is performed.

V. EXPERIMENTS AND RESULTS
We train and evaluate our proposed algorithm, the QADRA scheduler, in a full scale NR system level simulator. Network parameters used in all runs are shown in Table I.

A. Experimental setup
We deploy our algorithm in a scenario with the following traffic model: a single full buffer downlink operating simultaneously with ten VoIP users, each of which comprises an uplink and a downlink data flow. All UE have relatively good channel conditions (see Table I). The particular scenario is prone to Algorithm 3: Learner process Data: Recurrent network h with two sets of params φ, θ. Feed-forward network Q with params ψ. Learning rate η ∈ R. Let Γ be short hand for {φ, θ, ψ}. Also, let s be the concatenation of s i and s o 1 Fixate φ − , θ − , ψ − ← φ, θ, ψ; 2 while training do 3 Sample (X τ , a τ,1 , . . . , a τ,Nτ , r τ , X τ +1 ) from the replay buffer; 4 Reset states: Reset gradient: ∇ Γ ← 0; Occasionally: re-fixate φ − , θ − , ψ − ← φ, θ, ψ; 25 end   resource starvation (cf. [8]) with increasing number of VoIP users. This is shown in Fig 4 with a round robin heuristic, i.e. data flows are given a weight proportional to their timesince-last-scheduled [1], [3]. We compare the downlink network throughput and the VoIP user QoS satisfaction rate achieved using QADRA to those achieved using the round-robin and proportional fair heuristics [3]. The proportional fair heuristic is defined as w j = ū u , where u andū denote the expected throughput and mean throughput, respectively. Finally, in these baselines we also prioritize re-transmissions, however these are rare due to the cellular reception quality.
As defined in Sec. IV, the reward is measured in terms of the (downlink) throughput of the full buffer traffic, and the QoS satisfaction of VoIP data packets through the two-dimensional vector r = [r fb , r VoIP ] T . We define the QoS requirements of VoIP data flows as a maximum packet delay of 100 ms 1 . The reward function for VoIP data packets is therefore given by r VoIP (d) = − 1{d > 0.1}, where d is the packet delay measured in seconds. Further, we let r fb equal the number of bits successfully transferred over the network by the full buffer in the current TTI.
We let the feature space of data flows consist of the following six variables: (1) time since last scheduled, (2) traffic type (encoded as an integer), (3) number of bits ready for transmission, (4) flag indicating up or downlink flow, (5) flag indicating new transmission, and (6) flag indicating re-transmission. Each data flow is therefore characterized by a feature vector in Each experiment was run using four actor processes gathering experiences to the learner process. Every 10th TTI, the network parameters and exploration rate are updated from the learner process. Inspired by [9], we vary the exploration rate throughout training. In particular, we let the exploration rate of the pth update be given by p = a 1+up , where u p = p mod 8. The value of a decreases exponentially from 1.0 to 0.4 throughout training. During the first 20000 TTIs of training, we let = 1 in order to build the replay buffer, which has a capacity of 131072 sequences. The training process is only started when the buffer reaches 20000 sequences. Before starting, the mean and standard deviation of the feature variables are computed and then used for normalization of inputs to the networks. Finally, 1   in addition to the DQN, we also implement the double [10] and prioritized experience replay [11] DQN extensions. The recurrent networks consist of a fully connected feedforward network with ReLU activations and layer sizes 6 × 256 × 128, followed by three gated recurrent units [12] with state sizes 64 × 32 × 32. The final state is used as the output, i.e. each list is encoded into R 32 . The fully connected feedforward Q-Network uses layer sizes (6+32+32)×512×256× 128 × 64 × 1 and ReLU activations. The learner process uses an Adam optimizer [13] with a batch size of 32 and learning rate 10 −4 . Moreover, we normalize the rewards by factors 200000 and 0.01, for the full buffer and VoIP rewards respectivelyhowever these constants can of course be seen as part of the preference vector, ω.

B. Results
Next, we present the numerical results obtained through our experiments. All results are obtained by running the trained agent greedily, i.e. = 0, for 30 simulation seconds. In Fig. 5, we present empirical distributions of (a) downlink   Table II. In particular, the DRL configuration ω T = [10, 0.1] results in approximately 30% improved downlink network throughput with no impact (minor improvement) on the QoS satisfaction rate of VoIP data packets. Compared to round robin (black curve), throughput for this configuration is increased by approximately 20% while QoS-affecting delayed packets are reduced by 30%. Furthermore, from Table II and Fig. 5, we observe that tuning the preference vector provides fine-grained control over the scheduler QoS performance. Specifically, an increased weight on VoIP improves QoS performance at the cost of downlink throughput and vice versa.

VI. CONCLUSIONS
In this paper, we have proposed a new approach to time domain scheduling in cellular networks using deep reinforcement learning. In contrast to current state-of-the-art algorithms, sorting data flows implicitly through heuristics, our proposed algorithm explicitly sorts data flows through a selection sortlike process. Moreover, we train our algorithm end-to-end as to explicitly optimize user experience and network performance, thus limiting the need for human interference and/or manual tuning of the implemented scheduler. Finally, we allow a network operator to control fundamental network performance trade-offs, explicitly and reliably, through the preference vector.
We numerically evaluate our scheduling algorithm in a full scale, near-real, NR system level simulator, and compare it to two state-of-the-art scheduling algorithms. Specifically, we consider a network of ten VoIP users and one full buffer downlink flow. The results demonstrate that our scheduling approach provides a significant boost in network performance. This is illustrated with an increased cell throughput of approximately 30% while simultaneously keeping QoS of VoIP data flows unaffected (see Table II).

A. Future work
We consider two directions for future work. Firstly, although evaluated in a rather specific resource starvation scenario, we believe that our algorithm is capable of learning optimal scheduling in more complex network scenarios as well, involving increased number of data flow types and users. Secondly, extending the algorithm to consider varying optimization objectives may allow for varying preference vectors at inference time, thus more control of fundamental trade-offs. Incorporating the work of Yang & Sun [14] into the algorithm may yield interesting results.