{'fraction_non_alphanumeric': 0.03373203996790898, 'fraction_numerical': 0.005360659324629859, 'mean_word_length': 4.938284971849286, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
From Design to Deployment of Zero-touch Deep Reinforcement Learning WLANs

Machine learning (ML) is increasingly used to automate networking tasks, in a paradigm known as zero-touch network and service management (ZSM). In particular, Deep Reinforcement Learning (DRL) techniques have recently gathered much attention for their ability to learn taking complex decisions in different fields. In the ZSM context, DRL is an appealing candidate for tasks such as dynamic resource allocation, that is generally formulated as hard optimization problems. At the same time, successful training and deployment of DRL agents in real-world scenarios faces a number of challenges that we outline and address in this paper. Tackling the case of Wireless Local Area Network (WLAN) radio resource management, we report guidelines that extend to other usecases and more general contexts.


I. INTRODUCTION
WLAN has become the ubiquitous access technology at home, in public locations such as train stations, or private ones such as university or corporate campuses. Especially in densely populated areas, scarcity of radio resources can easily lead to congestion and thus bad user experience. Luckily, the fleets of WLAN access points (APs) in campus networks can be centrally controlled, opening the way for dynamic and autonomous configuration of network resources: as many of such dynamic resource allocation problems are hard, they are solved in practice using well-thought heuristics.
Inspired by success of Machine Learning (ML), the field of communication networks has been actively seeking to exploit such techniques to automate complex network tasks, paving the way toward the realization of zero-touch network and service management. In particular, Deep Reinforcement Learning (DRL) techniques, which learn by interacting with an environment, are able to achieve complex tasks with unprecedented skills -top stories include Google's AlphaGo [1] beating the Go world champion Lee Sedol in 2016, or OpenAI Five [2] winning an online computergame DOTA2 tournament in 2017, or recent advances in fully autonomous cars from Tesla [3]. Following similar path, recent attempts to use DRL instead of heuristics for automating network resource allocation [4], routing [5], WLANs configuration [6] and more [7]- [9] have shown promising results. At the same time, we observe that while it is relatively straightforward to design and train DRL agents that work well in synthetic and controlled settings, realworld deployment of the same DRL agents poses a set of additional challenges. Indeed, performance evaluation in simplified settings remains a necessary task (i.e., if a solution does not work in simulation, it will never work in the real world), but it is clearly not sufficient (i.e., there are no guarantees that the DRL solution will work as expected in a different environment than the ones on which it was trained). Thus, in order to carry DRL all the way from design to deployment, a number of practical and often underestimated challenges must be accounted for. The latter are just as important as the ML algorithmic design.
Such challenges are rooted in the architectural requirements that must be fulfilled in order for ML techniques to be seamlessly applied in the network: these are nailed down by standardization bodies, as for instance ETSI Zero-touch Network and Service Management (ZSM) 1 , that provide normative architectural references for several tasks. Clearly, as ML techniques are data-driven, a set of requirements concern access to telemetry data, notably the ability to stream it (ZSM requirement #84), enforce access control (#86) and store it in data lakes (#87). In particular, DRL model training requires access to data lakes (and likely GPU resources), whereas DRL model inference requires access to stream telemetry (and significantly less computational resources). Furthermore, ML operation requires the ability to deploy and upgrade trained ML models (#46, #49): while some "default" trained model may be necessary for generic zero-touch operation, the same model may be "fine-tuned" to the specifics of the environment after deployment. Training a generic model requires historical data gathered from several networks and available in a data lake, while model upgrading requires fresh telemetry for the purpose of fine-tuning. Finally, and most importantly, closed-loop techniques such as DRL need the ability to enforce actions automatically (#68, #115), depending on specific conditions determined by the algorithm, in order to adapt resources allocation better to the instantaneous or forecast evolution of service load.
This paper reports our experience in designing and deploying DRL for zero-touch WLAN networks. We build over our original design of a DRL sequence-to-sequence architecture, that we limitedly validated for the purpose of WLAN resource allocation in simulated settings [6], and that have now been running for months on real operational deployments. In the path from design to deployment, we outline and tackle five important challenges related to (i) safety, (ii) duration and (iii) realism of the training process, as well as the (iv) generalization capabilities and (v) the adoption barrier of trained models. In sharing our experience with the community, we not only illustrate the specific way in which we overcome such challenges in the WLAN case, but further adopt a broader viewpoint: we complement lessons learned with those gathered from other fields where DRL has been successful, such as gaming [1], [2] and self-driving cars [3], [10], testifying the generality of these challenges.
The rest of the paper presents a high-level view of the zerotouch WLAN resource management problem ( §II), articulates the main challenges from design to deployment ( §III), and summarizes the main lessons ( §IV).

II. ZERO-TOUCH DEEP REINFORCEMENT WLANS
Our goal is to autonomously manage WLANs in closedloop, continuously adapting allocated radio resources to changing traffic conditions and demand, to maximize end-toend performance. We first briefly cover WLAN management ( §II-A), that we next reconsider under the lens of zero-touch closed-loop control ( §II-B), and finally overview the DRL technique we employ ( §II-C).

A. Wireless LANs
WLANs are defined in the IEEE 802.11 standard 2 and its amendments. Zero-touch operation in the general case [8] and in heterogeneous, industrial and enterprise WLANs are surveyed in [9]. Here we provide a very basic overview of WLAN resources and actions from the viewpoint of autonomous closed-loop control. The most popular WLAN setup is infrastructure-based, where stations (such as smartphones, laptops or industrial devices, referred to as STAs) connect to fixed Access Points (APs) that typically act as gateways to relay STA traffic. In office buildings or university campuses, a fleet of APs is deployed over a large area to connect the numerous STAs to the Internet. Typically, centralized management decisions are taken to optimally manage the network: the set of actions include 3 , for each AP, selecting a channel, bonding and power configuration.
Each AP is configured to use a specific primary channel (within the 2.4GHZ or the 5GHz band), performing downlink and uplink transmissions in a half-duplex manner. Optionally, an AP may be configured to allow the aggregation (aka bonding) of several channels, to increase bandwidth and consequently throughput. Ideally, only one device within receiver vicinity transmits on one channel at the same time, avoiding collisions and data loss. This is achieved through the listen-before-talk mechanism of carrier-sense multiple access with collision avoidance (CSMA/CA). As a consequence, APs and STAs on the same channel share airtime: the time that a transmitter waits while the channel is busy is called interference (time). Depending on the regulatory region, only 4 (20) channels are non-overlapping on the 2.4 GHz (5 GHz) band: thus, it is not always possible to allocate different channels to neighboring APs and, in highly dense areas, interference cannot be avoided by simple channel allocation. It follows that on top of channel allocation and bonding [11], also the AP transmit power [12], [13] can be additionally used to tradeoff the strength and quality of the received signal vs the airtime interference.

B. Zero-touch WLANs
We note that the network configuration has to be selected from a very large state space, that grows exponentially with the set of available configuration knobs. Additionally, as the network load evolves over time, it would be desirable for the WLAN to be able to autonomously adjust its configuration to best adapt the available resource to the current (or forecast future) demand. As the utility function to estimate network quality can be a complex combination of QoS (e.g., signal strength, coverage, interference, user throughput, latency) and QoE indicators (e.g. more advanced per-application metrics), this makes autonomous configuration a desirable capability of WLAN networks. From this viewpoint, with reference to Fig. 1, it is envisionable that a zero-touch WLAN is governed by an ML model taking decisions (e.g. configuration actions) as a function of observable state (e.g. stream telemetry). Such a model should be pre-trained (e.g., by using a digital replica 4 of the network), but could possibly benefit from specific finetuning from real-data after deployment (to upgrade the model in the long run).
We observe that the existence of two separate environments results in a dichotomy inducing three separate loops: a first design/train/validation cycle (left of Fig. 1) where the ML model is trained on a digital replica of the network; a second deploy/inference/test cycle (right of Fig. 1) where the trained model is used on the actual network; a third refinement loop, bridging the two environments. The picture also highlights several important practical aspects that this paper is going to dissect, notably: pre-training is necessary to 1 train safely and 2 train fast, but note the need for 3 environmental realism for better fit and 4 generalization capabilities to unknown The critic learns the value function, which is used to "criticize" the actor and guide its learning.

Value estimator Neural Network
Multiple actions (run actor multiple times) Q estimator Neural Network select best action Environment Regret The selector learns the Q function to select the best action among many proposed by the actor Fig. 2. Simplified synoptic of the DRL architecture for WLAN management, described in more details in our prior work [6].
states. Finally, 5 explainability and trust are key to deploy zero-touch closed-loop operation.

C. Deep Reinforcement WLANs
DRL techniques are suitable for implementing the closedloop control algorithm. To better understand challenges that DRL agents may face in real-world deployment, we first briefly remind its most important concepts, and cast them to WLAN with the help of Fig. 1. Without loss of generality, in the the context of this work we limit the configuration knobs to the the selection of primary channel and bonding.
1) Background on RL and DRL: In Reinforcement Learning (RL) [14], an agent learns from interacting with an environment: the agent obtains a perception of the environment through a measurable state (e.g., network configuration, stream telemetry, etc.) and selects an action (e.g., changing an AP configuration) based on a policy that it is learning. After enforcing the action, the agent observes an updated state and receives feedback about its action, in the form of a reward (or a regret). Unlike in supervised learning (where the feedback reflects some distance from the optimum), the feedback in RL can be rather seen as praise (or critique) of the action (without any explicit information about the optimality). Based on this feedback, the agent updates its policy, observes the new environment state and enforces a new action, learning to increase its reward (or decrease its regret). Deep Reinforcement Learning (DRL) is a class of approaches that are based on a Neural Network (NN), where the NN is used either to learn the value of the state (i.e. DQN) or the policy (e.g. A2C, A3C). A recent wave of DRL approaches have shown interesting results in the solution of combinatorial graph problems, using various architectures, such as Graph Neural Networks, pointer networks and graph attention networks [7].
2) Overview of WLAN DRL agent: In our previous work [6], we used a similar philosophy to develop a DRL architecture fit for WLAN channel management. While in [6] we limitedly validate the approach against state of the art via simulation (left of Fig. 1), in this paper we are concerned about the complementary necessary steps for real-world deployment (bridging the gap between left and right of Fig. 1). As such, we provide here only a necessary limited overview of the WLAN DRL in Fig. 2, and refer the reader to [6] for details.
Our design follows a classic actor-critic Neural Network (NN) architecture, where the critic-NN which learns the value function guides the actor-NN in learning the best policy, to which we add a selector-NN, guiding the choice of the best action among those returned by multiple parallel runs of the actor-NN. The actor-NN employs an encoderdecoder sequential architecture, where basically the same NN is run sequentially multiple times, picking at each step a decision for one of the APs in the network. In particular, our encoder-NN employs a careful feature engineering process to transform variable-size input features (which depend on network size, number of channels etc.) into a fixed-size intermediate representation, that is used as input by the decoder-NN to output a probability distribution over all actions (i.e. one channel and bandwidth option for each AP), which makes it suitable for application to arbitrary networks.

III. FROM DESIGN TO DEPLOYMENT
We now report on our deployment experience of DRL agents in real operational WLANs. In particular, our DRL-based WLAN channel management solution (agent) autonomously reconfigures in closed-loop (action) every 10 minutes (so that the need for accurate forecast of future demand is lessened by the fact that actions are frequently taken) a real operational WLAN (environment), based on telemetry data (state) received at sub-minute timescale.
While training and deploying this DRL-based system, we faced a series of challenges 1 -5 mentioned in Fig. 1, into which we now dig deeper -first summarizing our experience with WLAN deployment, and next contrasting the lessons learned to other DRL real-world use-cases.
A. Train safely 1) WLAN insights: Training on the real WLAN network would inevitably lead to the exploration of bad network configurations harming user experience. As this option is not viable for business considerations, and while an expensive WLAN testbed is not available, we are forced to train the DRL agent using a digital replica of the system, such as a computersimulated model of the real environment. While learning from a digital "twin" solves the safety concerns altogether, it does however introduce another tradeoff. Namely, the simulator needs to be realistic enough to favor the transfer of learning to the real deployment ( §III-C) and at the same time simple enough to allow for reasonable training time ( §III-B), which are both key aspects that are worth digging into deeper.  2) Beyond WLAN: In autonomous driving, the necessity to training safely is even more obvious. For example, AWS deepracer [10], a platform to train and test DRL agent for this usecase, relies on a cloud based 3D racing simulator as one key component that helps avoiding the exploration of the most detrimental states even with model cars. Tesla also trains its Autopilot offline before allowing it on the road [3].
B. Train fast 1) WLAN insights: Two factors impact the training duration of our agent: (i) the convergence of DRL weights during the training process, which affects the number of interactions, and (ii) the duration of each simulated interaction, during which the DRL training process remains idle waiting to receive state and regret feedback from the environment. As for (i), we calibrated the training phase carefully to avoid getting stuck in local minima, mainly by adjusting the learning rate during training such that more aggressive updates (e.g. larger steps) are performed at the beginning, followed by smaller steps allowing the system to gradually stabilize. As for (ii), the duration of a simulated interaction can quickly become a bottleneck, for which we rule out the use of packet-level simulations (such as ns-2 or ns-3) and leverage a fast custom simulator, with low computational complexity. Fig. 3 illustrates the regret evolution over multiple independent training runs: the x-axis reports the number of iterations, the GPU training time (including the simulation time, measured in hours) and the equivalent duration of the training process had it been performed at the same timescale in a real environment (measured in years).
2) Beyond WLAN: Training duration is a clear bottleneck in any DRL deployment. In the most recent successful DRL applications, agents need several "lifetimes" of interaction with the environment, e.g. the 10,000 years equivalent of gameplay for OpenAI Five [2], which is clearly unrealistic for training on real systems. Even training offline with real data can take a significant amount of time: for instance, it takes 70,000 GPU hours to train the full self-driving Tesla pilot prototype [3], which is around one year for a single node with 8 GPUs. In some cases, training needs to be offloaded to a large fleet of data center servers equipped with GPUs and TPUs, which may be only affordable for a few big players. in network usecases, depending on the size of the DRL NN, the digital "twin" can become the computational bottleneck.
C. Environment realism 1) WLAN insights: As environment realism remains a key concern, we can leverage real-world data to enhance the simulator models -briding the training and validation environments. To retain scalability while enhancing realism, we use data-driven approach to refine several models used in the simulator. To make just a single example in reason of space limits, our simulator uses a RSSI threshold to decide which APs are considered as neighbors: we increase realism by fitting this parameter to maximize similarity between the interference estimations of the simulator and those measured in the real network. As can be observed in Fig. 4, the ideal RSSI threshold under which two APs should be considered neighbors is at -82dBm, with which we calibrate the simulator for fine-tuning the DRL agent.
2) Beyond WLAN: Environment realism clearly is key when an agent trained on simulation is to be transferred into the real world. When training on a simulator, sufficient resources need to be invested into its calibration [10]. We point out that alternatives to simulation exist, which may however not be fit to network use-cases. For instance, imitation learning is a valid complementary approach, where instead of interacting with a real or simulated environment, a database of state-action traces is used for offline batch-based agent training. For instance, thanks to its fleet of several hundred thousand self-drivingready cars, Tesla now has a huge amount of state-action pairs (over 1 billion miles with Autopilot-on [3]), which might be used to learn to mimic human driver behavior.  However, data-driven approaches in general, and in networks in particular, may be vulnerable to data scarcity and quality issues. For instance, while in WLAN we have access to live measurements and long historical data from real networks, since the network is sparsely reconfigured (once per day in legacy data), our datalake is limited in terms of number of explored states, which DRL would require for training. Additionally, the states explored currently (i.e., network configurations) are limited to the subset induced by the existing algorithm, further restricting the boundaries of what DRL could possibly learn. Thus, simulations, augmented with data-driven models, are a better option.
D. Generalization 1) WLAN insights: Generalization to conditions unseen during training means, on the one hand (i) generalizing to WLAN networks of arbitrary size and density, and on the other hand (ii) transferring well to the more complex physics of the real network -which are both necessary as deployment conditions will never match exactly the training conditions. We tackle (i) by a novel auto-regressive sequential decoder whose input features at each step are engineered to reflect the changing internal state of the decoder that is described in [6].
As for (ii), robustness of the DRL agent to varying conditions is key: to test the ability of the trained DRL model to cope with unknown conditions, we train the agent in an ideal environment and test it in a noisy one. We systematically apply Gaussian noise with controlled means and standard deviations to the AP neighborhood (i.e., RSSIs each AP sees from all others) and observe the impact on the regret. The left-hand side of Fig. 5 reports the relative percentual increase of the regret with respect to ideal conditions (noiseless training and testing) when neighborhood is defined with a simple threshold. Overall, the picture confirms DRL to be robust for a wide range of additive noise. Additionally, consider that a negative noise causes the corresponding RSSI to fall below the neighborhood threshold, leading to interference underestimation. Unsurprisingly, the figure confirms underestimation to be more harmful than overestimation, which validates our conservative simulator design choice.
However, in the real network, neighborhood is not exactly a clear-cut threshold: the right-hand side of Fig. 5 further tests the algorithm on the same noisy conditions, but using a more complex neighborhood definition: in particular, neighborhood interference is smoothly taken into account by using an S-shaped sigmoid function with a spread of 6dB centered around the clear-cut threshold. Interestingly, using this "loose" definition does not degrade the results and leads to even better resistance to noise, suggesting good generalization ability.
2) Beyond WLAN: In any DRL deployment, when digital twins are used for training, realism of the simulation is of primary concern -the less faithful the simulator, the lower the quality of the agent. In our case, we analyze the response to the trained model by stress-testing it against environmental changes. We point out that (i) synthetic noise or (ii) real-world data can be readily incorporated during the training process, although the exact way to do so depend on the specific usecase.
E. Explainability and Trust 1) WLAN insights: Lastly, it is essential that network engineers develop trust in the algorithm decisions before letting it run unattended on thousands of customer deployments. As DRL decisions are intrinsically less interpretable than heuristics, this can firstly be achieved by human-understandable explanations of the algorithm decisions and expected gains -so that the WLAN operator can not only perceive DRL operation as safe, but also understand and value its benefits.
Additionally, we conducted months worth of tests in operational WLAN networks to illustrate the DRL agent viability. In particular, we ran several batches of experiments, each lasting 1 week, in which we either run the (i) trained DRL agent every 10 minutes to closely track load changes vs (ii) a daily static optimization based on historical load forecast. Here, we show an except of deployment results in a Campus network in Nanjing, China: Fig. 6 (left) contrasts the channel utilization on a 30 AP WLAN where we observe several thousand STAs on a typical day. We construct a heatmap from the scatter plot where each point represents the average channel utilization for the same AP during 10 minutes at the same time-of-day and day-of-week for the two algorithms, over all APs: this allows to assess the impact of dynamic DRL channel management from a spatial viewpoint, i.e. from the point of view of the same AP. We can note the tendency of improvements as the center for the highest density moves above the diagonal.
Clearly, while the traffic is similar in every week due to seasonal behavior of the users, the traffic conditions are not identical, which can bias the comparison. Trust in the solution can be then gained only over a careful analysis over long-term campaigns. For instance, we take this confounding factor into account by comparing in Fig. 6 (right) the breakdown of the AP utilization (y-axis) for the same average network load (xaxis): it is easy to see that, as expected, DRL relieves APs with high channel utilization (notice the 95th percentile decrease) by shifting load to lightly loaded APs (notice the median increase), which is desirable from the perspectives of load balancing and fairness. Overall, thorough testing under realworld conditions delivers a strong argument to IT engineers to trust the DRL agent for network O&M.
2) Beyond WLAN: As a general lesson, human operators need to gain understanding and build trust in DRL systems to allow their deployment. Explainability of the algorithm output helps lowering the adoption barrier. Trust may then be gained step-by-step: convincing results from extended experiments on real-world deployment help showing the benefits of the algorithm -which holds for any use-case.
For instance, Google DeepMind adopted the following strategy for data center cooling: the first RL algorithm version 5 acted only as a recommendation engine for the operator. Only later they moved to a fully autonomous version 6 , still maintaining a failsafe option to revert back to human control at any time, in addition to rule-based heuristics as backup.

IV. CONCLUSION
DRL is a promising paradigm for controlling complex systems, improving decision making over human intuition and classic heuristics. While most network-related DRL research focuses on ideal scenarios and are evaluated via simulation, we discuss here the challenges that arise when DRL is deployed in large-scale operational WLAN to achieve zerotouch operation.
We generalize and summarize the lessons learned as follows. It appears that DRL training requires digital "twins", such as simulators. Indeed, solely learning from existing network data may not be feasible (given the sheer number of samples needed for training) nor desirable (as it does not offer enough action diversity, so missing unsafe actions). Conversely, learning from simulation provides the best tradeoff among safety (i.e. to explore also unsafe actions), simplicity (for training duration) and realism (e.g. as simulators can be enhanced with real-world data and be used to assess controlled generalization). Finally, technical benefits are a necessary (but not sufficient) condition to adoption: deployment of trained DRL models for real-time inference still requires a pedagogic effort toward the human operators interacting with it (in order to gain their trust), as well as offering fallbacks to legacy systems until the algorithm gains sufficient trust for true fullyautomated zero-touch operation.