{'fraction_non_alphanumeric': 0.03146633102580239, 'fraction_numerical': 0.005506607929515419, 'mean_word_length': 4.785497572815534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Reinforcement Model Selection for Communications Resource Allocation in On-Site Medical Care

Greater capabilities of mobile communications technology enable interconnection of on-site medical care at a scale previously unavailable. However, embedding such critical, demanding tasks into the already complex infrastructure of mobile communications proves challenging. This paper explores a resource allocation scenario where a scheduler must balance mixed performance metrics among connected users. To fulfill this resource allocation task, we present a scheduler that adaptively switches between different model-based scheduling algorithms. We make use of a deep Q-Network to learn the benefit of selecting a scheduling paradigm for a given situation, combining advantages from model-driven and data-driven approaches. The resulting ensemble scheduler is able to combine its constituent algorithms to maximize a sum-utility cost function while ensuring performance on designated high-priority users.


I. INTRODUCTION
The optimal allocation of communication resources is part of the perpetual race for better performance in mobile communications. As new technologies such as high resolution video streaming and vehicular communications emerge, the demands to be fulfilled by a scheduler are becoming increasingly heterogeneous, resulting in complex optimization tasks that must be solved in real time.
In recent years, deep learning (DL) methods have distinguished themselves with success in complex tasks, including image classification [1] and control [2], among others. DL, as a subset of machine learning (ML), follows a different paradigm compared to classic model-based approaches. Instead of designing an optimal algorithm for a given goal, an algorithm is iteratively approximated based on training data. Without the need to explicitly model the This work was accepted for presentation at IEEE WCNC2022. 1 underlying processes, the issue of modeling complexity is sidestepped. This characteristic has sparked an interest in research that applies DL to the highly performance-driven field of communication systems.
In particular, increasing the performance and capabilities of mobile communication systems creates new prospects in emergency patient care. For example, video, vitals, or specialist input can be exchanged wirelessly between on-site medical professionals and hospital staff, yielding a significant head start in time-sensitive treatments [3].
Nevertheless, a deliberate approach is required due to the low tolerance for delays and outliers in medical tasks.
Various forays have been made into integrating DL methods with resource allocation tasks, mostly based on deep Q-Networks (DQN) [4]- [6] and actor-critic methods [7]. However, model-based approaches have long thrived, thanks to the valuable expert knowledge that shapes their design. Ideally, incorporating this expert knowledge into data-based methods could govern and stabilize the learning process and produce more tractable algorithms [8].
Therefore, we investigate a combined approach that offers some of the advantages of both DL and model-based approaches. For the task of scheduling discrete resources in vehicle-to-base-station communication, we implement a group of simple, model-based scheduling algorithms. We then train a DQN to select the best model-based algorithm in a given situation, optimizing the long term effect on typical performance indicators: time-outs, sum capacity, and packet rate. Particular attention is paid to the performance of Emergency Vehicles (EV) as a priority class of users.
In this way, we are able to use data-driven DL to find an approximately optimal solution to a complex scheduling problem while still using explicitly modeled algorithms.

II. SETUP & NOTATION
In the following, we introduce our system model. We describe the communication channel between a user and the base station, the resources available for allocation, and the generation of jobs. We then formulate the resource allocation problem. After these technical specifications, we move on to the DQN scheduler design.

A. Resource Allocation System Model
In our system model, a number of N user vehicles are connected to a base station, moving on a 2D plane in a grid akin to the Manhattan-model of movement [9]. At any discrete time instance t, vehicles n will take a fixed-size step in a direction of movement that is selected randomly. In doing so, the vehicles have a 98 % chance of re-selecting their prior direction of movement, 0 % chance of selecting the direction opposite to their prior movement, and uniform chance of selecting a 90°turn left, right, or stopping.
The communication channel between the base station and a user vehicle n is characterized by its power gain The power gain is known to the base station. For each simulation time step t, fading channel amplitudes |h n [t]| are randomly selected from a Rayleigh distribution and multiplied with a distance-proportional path loss factor path loss, we selected an exponent of −1 to reduce the spread of the path loss values encountered. For the purpose of this paper, this lessens the computation required in the subsequent learning task.
The base station has access to a limited number U of discrete resource blocks available for allocation, as shown in Fig. 1. These resources can be filled with jobs j from the job queue. At each time step t, new jobs are generated at a probability p j per user. A job j is assigned to a specific user n and is defined by two attributes, a request size u j, req [t] in discrete resource blocks, and a time-to-timeout v j [t] in discrete simulation steps. We define the set J[t] of all jobs in the job queue in time step t and the subset J n [t] of the jobs in this queue assigned to user n in t.
The job attributes' initial values are governed by a user profile that declares a maximum job size u n, max and initial time-to-timeout v n, init for users n of that profile. Upon generation, the job is assigned a size u j, req [t] selected from a discrete uniform distribution, u j, req [t] ∼ U{1, u n, max }. When a number of the base station's resources U is allocated to a job j, that job's remaining size is decreased accordingly, while a lifetime count u n, sx [t] of resources scheduled to user n is increased. The jobs' initial time-to-timeout v j [t] ← v n, init is decremented for each time step t that passes without the job being fully scheduled. Once the time-to-timeout v j [t] reaches zero, the job is discarded from the job queue and added to a set J fail [t] for that time step t, for use in performance metric calculation.

B. Problem Statement
A scheduler is tasked with assigning the limited number U of resource blocks to the jobs in queue. Three metrics are selected to gauge the performance of a scheduling algorithm: : Global channel capacity achieved Firstly, the global sum of resource blocks discarded due to timing out should be minimized.
In the desired RL loop, the parametrized scheduler interacts with the system by selecting an action A and receiving a resulting reward r[t] and new system state S[t + 1]. Based on these experiences, the scheduler updates its parametrization θ to promote high-reward actions and demote low reward actions.
Secondly, a packet rate is introduced as the lifetime ratio of resources requested and scheduled for a user n. We define the set J n, new [t] as the set of new jobs assigned to user n, generated in time step t. Using the lifetime sum of discrete resources u n, sx [t] scheduled to the jobs of user n, we calculate the sum packet rate over all users: A strong sum packet rate performance is achieved by a scheduler that does not neglect any single user.
Last, the sum rate capacity achieved by each transmission is calculated using the Signal-to-Noise ratio (SNR) resulting from the selected channels instantaneous fading characteristics as for a Gaussian input alphabet, where signal power P and expected noise power σ 2 noise are fixed for all vehicles n. The scheduler is tasked with balancing all performance metrics, thus, we collect all target metrics in a weighted sum utilityr with respective tunable weights w C , w P , w L . Additionally, we designate a priority class of EV-type users. Their significance is communicated to the optimization process by adding EV timeouts r L, EV [t] to the weighted sum utility with their own tunable weight w L, EV : III. ALGORITHM SELECTION APPROACH Where model-based approaches struggle to find an optimal solution, deep Reinforcement Learning (RL) may be applied to learn a function that approximates the optimal algorithm along the domain of reasonable input data. conditions to be met, which tend to be easy to formulate in model-based algorithms but cannot be enforced directly in standard RL approaches.
Our scheduler learns to adaptively switch between a selection of model-based algorithms to tackle this problem.
In order to select the best algorithm, the scheduler makes use of a deep Q-Network (DQN) [10] to learn the longterm benefit of selecting each operation mode in a given situation. As a result, model-based design benefits, such as hard performance guarantees and human interpretability, are provided by the pool of model-based algorithms.
Meanwhile, the superposition of algorithms allows for greater performance than each individual algorithm on flexible goal metrics. As an additional benefit, the individual model-based algorithms do not have to be sophisticated enough to perform well in every circumstance, so long as the overall selection is rich enough to serve any problem.

A. Model-Based Algorithms
Four model-based scheduling algorithms are implemented for the DQN to select from: [11] (1) A Maximum Throughput (MT) scheduler that allocates as many resources as requested by order of descending channel power gains. (3) A Delay Sensitive (DS) scheduler assigns a channel priority .
Each user is allotted a share of the available resources according to the normalized, weighted priority vector Uniquely, this scheduler skips jobs that are about to time out if the allotted discrete resources are not sufficient to complete the job. In this case, the resources are freed for the next highest priority.
(4) An EV Priority scheduler assigns as many resources as requested to any EVs and distributes remaining resources one-by-one, randomly assigning them to requesting users.

B. Deep Algorithm Selection
Our deep learning scheduler consists of multiple elements: A pre-and post-processor, a DQN, an arg max module, a memory module, and a learning algorithm that tunes the DQN. In this section, we will introduce these modules and illustrate their interconnection.
First, to be able to make informed decisions, a small pre-processor prepares information about the current state of job queue and communication link as a system state vector S[t]. Per user n, the pre-processor summarizes with reward r as defined in (7). The long term rewards are discounted by an exponential factor 0 ≤ λ ≤ 1. As depicted in Fig. 3, a DQN is set up to output an estimateQ of the long term expected rewards Q for each choice of model-based algorithm A i . Given a perfect approximationQ = Q, we maximize the long term expected rewards by simply selecting whichever action arg max iQ (S[t], A i ) has the highest estimated long term rewards. Therefore, the learning goal is to update the DQN parameters θ such that the estimate is approximately close to the true long term reward Q for all possible (S, A i ), relying on the universal approximation property of neural networks [10].
First, the scheduler must make experiences to learn from. Using an ǫ-greedy exploration scheme, the scheduler initially explores the simulation by taking random actions, i.e., selecting a random model-based algorithm for a given state. States, decisions, and their direct outcome are recorded and stored in a replay buffer in the form of We highlight that an experience only contains the immediate reward r[t], not the desired long-term rewards Q(S[t], A[t]), i.e., (8). However, we can extract additional information from S[t + 1], the state that following the action. By again using our DQN, we estimate the rewards following S[t + 1] to construct a learning target that incorporates the information of r[t] and S[t + 1] from the experience. Using this targetQ target (EXP), an estimation error [10] δ =Q target (EXP) −Q (EXP) (11) given the networks current parametrization can be calculated for any experience tuple from the buffer.
Following the principle of stochastic gradient descent (SGD), the networks parameters θ are then adjusted by sampling a mini batch of B experiences from the buffer to minimize a mean square error cost for the batch. Minibatch parameter updates are carried out every time a new experience is made.
As training progresses and the schedulers understanding of the simulation environment improves, the probability ǫ of selecting random exploration actions is gradually decreased, relying more and more on the network to make decisions. When prompted, at the beginning of a time step t, the network will estimate the expected long term rewardsQ for selecting any of the available model-based algorithms A i , given the current state S[t] and the networks current parameters θ. The scheduler then selects the model-based algorithm A i with the highest expected long term rewards.
To increase training efficiency, we implement optimizations to the base DQN learning method: • An infrequently updated network copy is used for the bootstrapping in (10) to increase training stability (Target

Network, [2])
• Sampling of experiences from the buffer is weighted proportional to the experiences' estimation error magnitude (Prioritized Replay, [12]) • The neural network structure is altered to seperately learn the contribution of state and action to the reward estimate (DuelingDQN, [12])

A. Implementation Details
We configure the simulation with U = 16 available resources and N = 10 users. User profiles are set up according to Table I, with five 'Normal', two 'High Datarate', two 'Low Latency' users, and one 'Emergency Vehicle'. For the given configuration, a job creation probability p j = 20 % for each simulation step and user puts a high expected load of 1.6 requests per available resource on average on the schedulers. We run a total of 10 000 episodes at 50 time steps t per episode for training and evaluation each, with the mini batch size set to B = 64 sampled experiences per training step. Parameter optimization via SGD is carried out by the Adam optimizer [13] with default settings and a learning rate of 1 × 10 −4 . Reward weightings are set to w C = w P = 0.25, w L = w L, EV = 1.0, giving roughly equal significance to each goal metric with the average expected magnitude of the respective rewards.
For the algorithm selection DQN, a five layer feed-forward network is selected. Layers have 300 nodes each, except for the last layer that was split in two branches with 200 nodes each according to the DuelingDQN structure [12].
Exploration is done by selecting random actions at an initial chance of ǫ = 99 %, decaying linearly to 0 % after 80 % of training episodes. The exponential future reward decay factor is set to λ = 0.9 in order to put significance on only a low number of future steps. User TX-SNR P/σ 2 noise is fixed to 13 dB. We implement the simulation in python using the tensorflow library primarily. The full implementation is made available in [14].

B. Results
As the simulation model contains stochastic components, the results achievable on each metric have an inherent variance depending on the specific realizations. For example, a spike in generated jobs will result in timeouts irrespective of the scheduling method. For this reason, results achieved during testing are displayed in cumulative histograms. Fig. 5 shows each schedulers performance on the combined reward metric r. On this metric, the DQN adaptive scheduler is able to find a strategy that outperforms any single model-based algorithm. Breaking the sum metric r down into its constituent parts sheds light on how this is achieved. As shown in Fig. 4, the DQN scheduler balances the submetrics against each other, yielding some performance on each of them compared to the benchmark.
Of particular interest are the EV-specific timeouts in Fig. 4d. This metric is not specifically targeted by any of the model-based algorithms depicted. A modest double-weighting of EV specific timeouts within the goal metric r, combined with the introduction of the otherwise sub-optimal EV Priority scheduling algorithm to the selection pool, has enabled the DQN based scheduler to significantly suppress timeouts in Emergency Vehicles even compared to the otherwise timeout focused Delay Sensitive algorithm without hurting overall performance overmuch. As Fig. 6 shows, the EV Priority algorithm was only selected a comparably low amount of times to achieve this goal.
The DQNs learned behavior can also be monitored to reveal underlying features of the simulation. As Fig. 6 highlights, for the given reward weighting, the MMF algorithm was only selected a low number of times. Further investigation could reveal whether these are, for example, high impact outlier cases that could be better served with an additional model-based algorithm that is specifically targeted to them, or whether the MMF algorithm is not fit for the given scenario.
The DQN decision making does however add another layer of computation to the scheduling process. Further, while the ensemble method can relax the sophistication required from each model-based part of the ensemble, the composite scheduling function can only assume the function space spanned by the group of model-based algorithms.
In other words, the DQN adaptive method is unable to discover strategies that go beyond combining the available models. Determining whether the selection of model-based algorithms provided is rich enough to serve the problem therefore remains a burden on the designer.

V. CONCLUSION
In this paper we presented a RL-based communications resource scheduler that constructs an effective scheduling paradigm from an ensemble of model-based algorithms. We achieved this by learning to switch to whichever algorithm promises the highest expected long term benefit based on the current queue state. This approach combines the flexible goal optimization of RL methods with the rigid predictability of model-based algorithms. It is noteworthy for applications with complex, conflicting performance goals, where either a) strong models exist to cover parts of the problem, or b) explicit modeling is otherwise necessary, e.g., due to very low tolerance for outliers, such as the transmission of critical medical data. Using this approach unlocks the benefits of RL without abandoning explicit modeling. For the simulation presented, the adaptive model-switching scheduler was able to learn to outperform single, model-based algorithms on a weighted sum utility metric.