{'fraction_non_alphanumeric': 0.024368953226158434, 'fraction_numerical': 0.009311122426711282, 'mean_word_length': 4.764360587002097, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reinforcement Learning Algorithm for Traffic Steering in Heterogeneous Network

Heterogeneous radio access networks require efficient traffic steering methods to reach near-optimal results in order to maximize network capacity. This paper aims to propose a novel traffic steering algorithm for usage in HetNets, which utilizes a reinforcement learning algorithm in combination with an artificial neural network to maximize total user satisfaction in the simulated cellular network. The novel algorithm was compared with two reference algorithms using network simulation results. The results prove that the novel algorithm provides noticeably better efficiency in comparison with reference algorithms, especially in terms of the number of served users with limited frequency resources of the radio access network.


I. INTRODUCTION
Evolution of telecommunication technologies allows cellular network operators to use radio resources more efficiently and therefore improve service quality and serve more users. Introduction of each new generation of radio access network architecture and gradual implementation of the new technology leads to a situation where several radio access technologies coexist creating a heterogeneous network (HetNet) [1]. Additional heterogeneity in network architecture is caused by many base stations types being installed for different coverage scenarios. These include macro cells for large area 1 Copyright (c) 2021 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication. The final version of record is available at https://doi.org/10.1109/WiMob52687.2021.9606426 coverage and small cells (i.e. micro-, pico-, femtocells) for more throughput-demanding coverage or areas not covered by macro cells [2].
Such complex and diverse RAN architecture causes many challenges for the network operator, including backhaul provision for each base station, inter-RAT and intra-RAT interference mitigation and fine-tuning base station parameters to maximize its capacity [3]. One of many important aspects of HetNets optimization is traffic steering, i.e. load balancing [4]. It is the process of adaptive traffic allocation to different base stations (or in more generic approach -available radio resources), often with a certain priority set by the network operator, e.g. to maximize total network throughput.
As in heterogeneous networks (HetNets) one user can often be in range of many base stations, the load balancing algorithm must decide on which cell's radio resources should be allocated to the user. Basic traffic steering algorithms rely on a single criterion to make a decision on what cell should serve given user. This criterion may be current load percentages of in-range base stations or satisfaction of user's bit rate demand with radio resources available to be allocated by each base station. These approaches guarantee moderate efficiency at a low computational cost, but are far from optimal.
The key paper contribution of this paper is a proposal of an universal traffic steering algorithm for utilization in HetNets. The solution includes a dedicated data-processing flow that combines ANN inference and SARSA algorithm for ANN optimization to provide near-optimal traffic steering. The idea itself is inspired by work described in [5], but aims to provide more universal utility for all generations of radio access network, including 5G and beyond.

A. Problem statement and reference scenario
The key problem addressed in this paper deals with traffic steering in HetNet scenarios. In general, the goal of traffic steering algorithms is to serve users using radio access network's resources in a way that uses the resources most optimally in regard to a chosen criterion. Basic approaches mentioned in [5] include allocating radio resources of the least loaded cell (denoted hereafter as Classic Load Balancing, CLB) or allocating radio resources of the cell that provides best user satisfaction (Satisfaction-based Load Balancing, SLB). In our approach, we aim to assign users to the cells in order to maximize total user satisfaction.

B. Reinforcement Learning Load Balancing
In the proposed scheme, we steer the cellular traffic within the HetNet by utilizing a reinforcement learning scheme called SARSA for optimization of an ANN [6]. In turn, the ANN adapts its model to use the limited radio resources in a way that maximizes the total user satisfaction in given scenario. The name of the SARSA method comes from the components used to update the state-action value function estimate [6]: The update step applied in SARSA is defined as in (1) [6]. One can observe that, besides the components listed above, it utilizes a learning factor α, that determines how much the new data influences the current estimate, and a discount factor γ to discount influence from predicted state on the current estimate.
As shown in [6], the probability of the SARSA algorithm convergence to optimal policy of the agent is 100% if the -greedy policy is utilized. It means that for each environment state observation, the action taken by the agent is random with a probability of , otherwise it results from the agent's policy. This approach has been utilized in the proposed algorithm.
In the context of SARSA application for traffic steering in HetNets, the environment is constituted by the considered radio access network area together with its active users. Next, the observation of the environment is a set of decision criteria for a single user. The reward after performing an action by the agent is the user's satisfaction after being served. In our approach, the ANN works as the agent, and its policy is modified according to the reward received from the environment after each action taken. This means that the ANN's connection weights and biases are modified according to (1).
The probability of a random action resulting from the adopted policy of the -greedy agent has been implemented in such a way that with successive simulation episodes it decreases by a certain dec value, descending to zero. Such a mechanism is aimed at reducing the impact of random decisions, when the optimization of the agent's policy allows obtaining satisfactory results. Based on the trials of the RLLB method with various combinations of the SARSA method parameters, the following values of the parameters were decided: = 0.1, dec = 10 −5 , α = 0, 15, and γ = 0, 95.
In order for the ANN to be able to optimize its behaviour an enhanced set of criteria is processed. In particular, for each user-cell pair, a set of three parameters is used: current cell load, percentage of cell's available radio resources that would be used up upon serving the user, and an estimated remaining number of users to be handled by the cell. The last parameter may be difficult to determine, however, in a real scenario, the initial process of load balancing could utilize one of the reference algorithms. Then, the number of users handled by the reference method can be used as input for the RLLB method.
Regarding the ANN structure, the selection of the number of convolution layers and the number of channels in each layer was performed on the basis of the method efficiency tests for various layer structures. For a large number of layers or channels in each layer, the efficiency of the RLLB method did not improve in relation to a network with a less complex structure. With too few layers or channels in each layer, the RLLB method lead to random decisions and its performance did Finally, a decision was made on the structure of a neural network with three convolution layers with the number of channels 20, 10 and 1. Additionally, the ReLU activation function (i.e. a function that outputs the input directly if it is positive; otherwise, it outputs zero) was used in the first two layers, which significantly improved the stability of the network. Next, the Sigmoid activation function (i.e. a function transforming input data to values from 0 to 1 [7]) was used in the third convolution layer, as it allows to obtain values that are convenient for interpretation. Thus, the output values of the neural network can be treated as an assessment of the goodness of the allocation of resources of a given base station to the user expressed as a percentage. Final structure of the neural network utilized in the RLLB method is shown in Figure 1.
Values at the output of the last convolution layer are subject to additional filtering to exclude base stations with insufficient signal coverage in the position of the currently considered user (i.e. channel quality indicator or user satisfaction equal to 0). The algorithm handles the user using resources of the base station with index equal to the index of the highest value after filtering the output of the ANN.

A. Base stations and user distribution
Simulation scenario for (downlink) traffic steering algorithms' efficiency comparison includes an urban environment with five base stations, including both LTE-A and NR radio access technologies. In the center, there is an LTE-A macro cell which can transmit with a maximum power of 43 dBm; two LTE-A and two NR micro cells are deployed within its coverage range, creating the HetNet scenario. The transmit power of the former cells is 32 dBm, whereas for the latter -34 dBm. The operating frequencies for the LTE-A and NR cells are 2100 MHz and 3500 MHz, respectively.
Users are distributed randomly in range of each base station, i.e. 300 users in range of the macro cell and 60 users in range of each micro cell. Each user is assigned one of the three available user profiles (as defined in Tab. I) with specific probabilities.

B. Radio channel quality model
Standardized channel model, described in [8], was used to calculate line of sight (LOS) probabilities and pathloss for each user-cell pair. The model has been utilized for both LTE-A and NR cells, as it is declared viable for frequencies from 500 MHz to 100 GHz. As the considered simulation scenario includes an urban environment, UMa and UMi variants of the channel model are used for macro and micro cells, respectively.
Based on the calculated pathloss, transmitted power of the given cell and gains/losses related to base station's and user's equipment, a value of signal power received by the user is determined for each base station assuming the receiver noise sensitivity at -110 dBm. The SINR value is then compared with target values in the standardized CQI (e.g. [9]) to find an estimated code rate used for further effective bit rate calculations. In the simulation, 20 MHz bandwidth was considered with and NR, accordingly.

C. Traffic steering process
Each simulation episode includes an allocation of radio resources to each user. At the beginning of the episode, it is assumed that all base stations have all radio resources available, and that no user is yet served by the network. The handling order of users is random. When the resources of all base stations in the test scenario are exhausted, the episode ends. It is assumed that the allocation of resources takes place within the network controller which has information about the load of each base station, the parameters of the signals received by the user's equipment and the bitrate demand.

IV. SIMULATION RESULTS
To evaluate the efficiency of the proposed solution extensive computer simulations have been carried out. Per each method 30000 episodes have been considered to guarantee statistical reliability of the results. Each method's efficiency was measured against two statistics collected at the end of each episode: mean user satisfaction (MUS), and mean not-handled-user (NHU) count. The former one ranges from 0 to 1 and is calculated as the mean ratio of demanded to received bitrate among all users. The number of NHUs is determined at the end of the episode as the number of users with a satisfaction of 0.
Efficiency statistics for CLB, SLB and RLLB method are listed in Table II, where S av is the MUS after 30000 simulation episodes with σ S being the standard deviation of mean satisfaction values and N av is the mean NHU count after 30000 simulation episodes with σ N being the standard deviation of mean NHU counts. The CLB method provides the lowest MUS of 86%, with its standard deviation equal to 2%. The SLB method managed to achieve 100% MUS in every episode. The RLLB method is almost as efficient in this respect as the SLB method; it managed to achieve near 100% MUS, with its standard deviation ten times lower than in the CLB method. Next, in terms of mean NHU count, the SLB method is the least efficient, as it leaves almost 76 users not handled on average. The CLB method is capable of handling one user more on average. The most efficient method in this respect is the RLLB method, which manages to handle two users more than the SLB method on average. Moreover, the RLLB method manages to combine the best features of both reference methods, with almost 100% MUS as the SLB method and a relatively small NHU count as the CLB method.

V. CONCLUSION
The novel algorithm called Reinforcement Learning Load Balancing successfully uses an ANN trained with the SARSA algorithm for optimal traffic control in radio access networks. The method adjusts its decisions based on feedback from the network in form of user satisfaction observed during the simulation episode. Decisions made with use of the continuously optimized ANN model allow for achievement of a small number of NHUs and high satisfaction of the users served.