{'fraction_non_alphanumeric': 0.0366570848387352, 'fraction_numerical': 0.008148949075661278, 'mean_word_length': 4.671452786917938, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 6, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 29, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Reinforcement Learning for Feedback-Enabled Cyber Resilience

Digitization and remote connectivity have enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an essential tool that epitomizes the feedback architectures for cyber resilience. It allows the CRM to provide sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resilience and discuss cyber resilience against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. We introduce three application domains of CRMs: moving target defense, defensive cyber deception, and assistive human security technologies. The RL algorithms also have vulnerabilities themselves. We explain the three vulnerabilities of RL and present attack models where the attacker targets the information exchanged between the environment and the agent: the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. Lastly, we discuss the future challenges of RL for cyber security and resilience and emerging applications of RL-based CRMs.


Introduction
Recent attacks such as SolarWinds [1] and the ransomware attacks on the U.S. gas pipelines [2] have shown intensifying concern of cyber threats on industrial and government cyber systems. These unprecedented attacks had created significant disruptions in business and government operations. We have witnessed not only a surge in the number of attacks but also their increasing sophistication. Now many attacks can circumvent traditional methods such as intrusion detection systems, firewalls, and encryptions. Advanced Persistent Threats (APTs) [3] are one of such threats that are known for their stealthiness, intelligence, and persistence.
Protection against such attacks becomes increasingly challenging. An attacker has to know only one vulnerability to launch a successful attack. In contrast, a defender needs to prepare for all its vulnerabilities and the associated attacks to secure the system successfully. Among those exploitable vulnerabilities, many of them are unknown until an attack exploit them (known as the zero-day vulnerabilities) or known but not patched timely (known as the 1-day or N -day vulnerabilities due to the delay of 1 or more days between the disclosure time and the attack time). An exploit directed at a zero-day vulnerability is called a zero-day attack, which is exceptionally difficult, if not possible, for the defender to design a protection mechanism to prevent the system from. This information disadvantage for the defender has made perfect protection not only difficult to achieve but also cost-prohibitive.
Furthermore, many recent attacks have targeted government agencies and critical infrastructures. The attackers are often supported by a nation-state or state-sponsored group. They are equipped with sophisticated tools, can conduct meticulous research about the system, and afford a prolonged period of persistent disguise. In contrast, security is often a secondary or add-on concern when designing or operating the system. A defender may not have invested sufficient resources in the protection mechanism. The striking disparity in the resources puts the defender at another disadvantage.
Both information and resource disadvantages have made a mere reliance on cyber protection an insufficient way to safeguard our networks. There is a need to shift the focus from cyber protection to a new security paradigm. Cyber resilience offers such a perspective [4,5]. It accepts the reality that the attacker can be successful in their attacks and complements the imperfect protection with resilience mechanisms. A cyber-resilient system adapts to the known and unknown threats and adversities in real-time and strategically responds to them to maintain the critical functions and performance in the event of successful attacks.

P2R2 Cyber-Resilient Mechanism
A cyber-resilient mechanism (CRM) can be decomposed into four stages: Preparation, Protection, Response, and Recovery. We also call it a P2R2 CRM. The first stage of the CRM is preparation. At this stage, we assess the security risk of the cyber system and design appropriate security policies, such as the deployment of honeypots or deception mechanisms [6,7], training of the employees [8], proper configurations of detection systems [9], and the preparation of the backups and contingencies for recovery plans [10]. Preparation is often done offline and ahead of the real-time operations. Good preparation can help facilitate effective prevention and fast response to unanticipated scenarios in later stages.
The second stage is prevention. At this stage, we implement the designed security policies and protect the cyber systems. Some attacks can be easily detected and thwarted in real-time because of the meticulous preparation and design of the security policies. For example, consolidating moving targeting defense [11] into the communication protocols would make it harder for the attacker to map out the traffic patterns and consequently thwart the denial of service attack. However, despite this effort, there is still a probability for an attacker to become successful, especially for a highly resourceful and stealthy one. The four stages of a cyber-resilient mechanism: Preparation, Prevention, Response, and Recovery. An attacker launches a sequence of attacks at t 1 , t 2 , t 3 , t 4 . The cyber security performance decreases when an attack successfully penetrates the system at t 2 . The system partially restores its performance at t 4 .
The third stage of response is critical to defending against attacks when we fail to thwart them at the prevention stage. At this stage, we acquire the information of the footprint of the attacker and reconfigure the cyber system to minimize the further risk of the attack on the cyber system [12,13]. The response to attacks is delay-sensitive. The information acquisition and the reaction to the observables need to be fast and done before the attacker moves onto the next stage of its attack. A fast response mechanism would rely on the acquired information and the designs at the preparation stage. There would be a cost in usability and performance when we respond to the attacks. The optimal response mechanism is a result of studying such tradeoffs.
The fourth stage of the CRM is recovery. The goal of the recovery stage is to reduce the spill-over impact of an attack and restore the cyber system performance as much as possible. The response to attacks in real-time is often at the sacrifice of the performance of the cyber system. There is a need to maintain system's operation and gradually restore its functionality to normal while reacting to the attacks. Fig. 1 illustrates the four stages of a typical CRM over a timeline. Before the operation, the cyber system designs a protection mechanism by considering the anticipated and known attacks and the risks of its own system. The cyber system starts to operate at t 0 . At time t 1 , an attacker launches attack a 1 . As the cyber system prepares for it, this attack is successfully thwarted, and the system is protected until the attacker launches a new attack a 2 at time t 2 . The cyber system did not prepare for this attack. The attack accomplishes its goal (e.g., lateral movement or penetration of the system) and, as a result, the cyber risk increases or the cybersecurity performance drops. The cyber system needs to respond to the successful attack and the ensuing attack a 3 at time t 3 . After fast learning from attacker's footprint, the cyber system makes strategic decisions to reconfigure and adapt to the adversarial environment and gradually improves the security posture of the system. The adapted system becomes more robust to attacks. The ensuing attack a 4 at time t 4 can be thwarted by the adapted protection mechanism. The cyber system restores to its best-effort post-attack performance at t 4 .
From Fig. 1, one can measure the cyber resilience across the four stages by the time it takes from the first attack to the recovery, i.e., T = t 4 − t 2 . This time interval indicates how fast the response is [14,15]. The worst-case performance degradation between the interval t 2 and t 4 is M .
The gap between the restored performance and the initial or planned performance, i.e., D, shows the effectiveness of the resilience. The natural goal of the CRM is to minimize T, M, and D. It is also evident that these metrics not only depend on the cyber system itself but also the type of attacks that the adversary launches. An optimal design would require understanding what threats are preventable and what can be mitigated by resilience.

Feedback Architecture and Reinforcement Learning
One critical component of the P2R2 CRM for cyber resilience is the response mechanism, which creates a dynamic process that involves information acquisition, online decision-making, and security reconfiguration. These components are strongly interdependent. One way to characterize this interdependence is through a feedback loop, as illustrated in Fig. 2. The feedback structure Figure 2: Feedback structure that enables cyber resilience. At each iteration, the cyber-resilient mechanism senses the parameters (e.g., cost and state) of the cyber system and makes decisions to act on the cyber system to reduce the cyber risks.
provides a system and control approach to cyber resilience [16]. The cyber system can be viewed as a plant. The CRM collects information from the cyber system through sensing and observation. The acquired information is used to reason about the current situation and decide how the system should react. The CRM acts on the recommended decisions by reconfiguring the security parameters and updating the cyber systems. The reasoning for the optimal cyber response can be viewed as a controller. The security reconfiguration can be interpreted as the actuation of the cyber system. The control perspective provides a rich set of tools and concepts from control theory to design an optimal, robust, and adaptive CRM to improve cyber resilience.
One pivotal control framework to achieve an optimal, robust, and adaptive CRM is reinforcement learning (RL). Similar to classic control approaches, RL leverages the idea of feedback architecture to guarantee certain aspects of performance such as optimality and stability. However, RL approaches are distinguished from classic control approaches from several facets.
First, RL allows the CRM policies to adapt to the online observations without knowing the underlying dynamics. The CRM can only observe the state s and the cost c from the cyber system at each step and then choose action a to optimally improve cyber resilience. RL is particularly useful for cyber systems as their precise models are often unavailable, and the influence of the attacker makes it even harder to map out an accurate cyber system model. There are also many unknowns in the attack model and cyber systems that require online learning and adaptation. RL provides an appropriate framework to bridge the gap between control theory and cyber resilience.
Second, RL allows the CRM to have a more generic space that captures the activities in the cyber system. Classic control oftentimes deals with physical systems such as mechanical systems [17], electrical circuits [18], chemical systems [19] and biological systems [20]. The state space of such systems is mostly continuous and finite dimension represented by R n with n being a finite integer. Cyber systems are computer & communication systems integrated with physical systems whose state space is discrete or hybrid (discrete and continuous) in most cases. RL is versatile at dealing with different types of state space. This versatility enables CRM to capture the complex status of cyber systems. Besides, RL equipped with function approximation techniques copes well with large-scale cyber systems that contain a large number of elements or have high-dimension state spaces [21,22].
Third, RL requires different stages than the classic control approaches before being implemented in real systems. Specifically, the implementation of RL techniques undergoes three phases: the training phase, the testing phase, and the execution phase. In the training phase, the agent interacts with the environment or the simulator to obtain data and improves his/her policy using the obtained data. In the testing phase, we evaluate the performance of the improved policy. If the policy reaches optimality or a satisfactory level of sub-optimality, the learned policy (or the learned CRM in our case) will be put into use, and the implementation enters the execution phase. Classic control approaches go through four phases: the modeling phase, the design phase, the testing phase, and the execution phase. In the modeling phase, modelers with domain knowledge build a mathematical model for the underlying system. In classic control, the training phase is replaced by a design phase where experts with domain expertise design a policy/controller based on the model constructed for the cyber system. The designed policy is then tested and evaluated before being executed in real systems. The training phases consume an enormous amount of data to improve the learned CRM constantly. Much effort need to be spent obtaining the data through either simulation or interacting with the environment. Data-inefficiency needs to solved by combining modern RL techniques with domain expertise from cyber security experts.

System Vulnerabilities and RL-enabled Defense
The uncertainties in the cyber system are caused by not only the complexity in its configuration but also the incomplete information of the attack models. An attack model specifies the vulnerabilities an attacker would exploit on the attack surface of the cyber system. The design of the CRM pivots on the type of vulnerabilities that the defender aims provide resilience for if he cannot succeed in defending against attackers who exploit them. We categorize the vulnerabilities into three major types. The first one is the class of posture-related vulnerabilities that arise from the resource disadvantage of a defender. The defender is not able to prepare for all vulnerabilities on the attack surface and has to rely on CRM to overcome this disadvantage. The second one is the class of information-related vulnerabilities that result from the information asymmetry between an attacker and a defender. The attacker has more information about the defender than the defender has about the attacker. The CRM is useful to tilt the information asymmetry and allow the defender to create uncertainties for the attacker while gaining reconnaissance of the attacker. The third type of vulnerability is human-induced. Humans are the weakest link in cyber protection. Social engineering, human errors, and insider threats are common attack vectors that an attacker can use to gain access to cyber systems before further penetrating the systems and reaching the targeted assets. The CRM can monitor human performance and guide humans to reduce the risks of their behaviors.
We use three applications to elaborate the design of CRMs for the three major types of vulnerabilities. The first application designs moving target defense (MTD) for posture-related vulnerabilities that result from the defender's natural disadvantages. The MTD creates reconnaissance difficulties for attackers by introducing uncertainties into the cyber systems. The authors in [11] explore the capability of RL in changing the system configurations to create the most uncertainty for the attackers while minimizing the change cost, which leads to a trade-off between security and usability. Both attackers and defenders apply RL to learn the risk and update their policies.
The second application addresses the information-related vulnerabilities that arise from the defender's information disadvantage. Deception technologies, including honeypots and honeyfiles, have been used to reduce information asymmetry between attackers and defenders. The authors in [23] use RL to design the deception technologies to engage the attacker and obtain threat information. The goal is to extract the most threat intelligence from attacks while minimizing the risks of detection and evasion. The study leads to the security insight that compared to an immediate ejection after detection, it can be more beneficial to engage attackers in the honeypots to obtain threat intelligence.
The third application designs CRM to mitigate human-related vulnerabilities. Adversaries have exploited human inattention to launch social engineering and phishing attacks toward employees and users. Compared to the reactive attentional attacks that exploit the existing human attention patterns, proactive attentional attacks can strategically change the attention pattern of a human operator or a network administrator. The authors in [24,25] have identified a new type of attacks called Informational Denial-of-Service (IDoS) attacks that generate a large volume of feint attacks to overload human operators and hide actual attacks among feints. They create an RL-based CRM to learn the optimal alert and attention management strategy.

Reinforcement Learning in Adversarial Environment and Countermeasures
As we use RL to improve the resilience of the cyber system, an intelligent attacker can also seek to compromise or mislead the RL process so that the cyber system ends with a worse or vulnerable security configuration. In this case, RL enlarges the attack surface and creates opportunities for the attacker. Hence it is essential to safeguard the RL while using it to secure the cyber system. Section 5 presents several attack models that target at RL algorithms and discusses their impact and ways to secure them. RL agents learn a 'good' policy through interacting with the environment and exchanging key information such as the rewards, the state observations, and the control commands. The three exchanged signals, which can be exploited by the attacker, become the vulnerabilities of most RL-enable systems. To understand RL in an adversarial environment, the first is to understand the adversarial behaviors of the attackers. To do so, we need to craft the attack models, which specify the attacker's objective, the attacks available to the attacker, and the knowledge the attacker has. The second is to understand how different types of attacks affect the learning results of RL algorithms, which requires the assessment of the performance degradation of the RL-enabled systems. With the understanding of attack models and their impact on the learning results, we can develop countermeasures to mitigate the effect of the attacks and to protect RL-enabled systems from attacks.
We review the current studies on security of RL and focus on attacks on the three signals, i.e., attacks on the rewards, attacks on the sensors, and attacks on the actuators. We present a sequence of attack models that can achieve a significant impact on RL-enabled systems with only a tiny effort on attacking. Given that this branch of research is still in its infancy, we identify several research gaps that researchers from the learning community and the control community can help narrow.
One missing component is that most works focus on deceptive attacks that falsify the values of the feedback information. In contrast, few works study the DoS/jamming attacks on RL algorithms. Another missing component is the study of attacks on actuators and the defensive mechanisms against these attacks.

Notations and Organization of the Paper
Throughout the paper, we use calligraphic letter X to define a set and boldface letter X to denote a vector. Notation E represent expectation and R + represents the set of positive real numbers. The notations Z ≥0 and Z >0 represent the set of non-negative and positive integers, respectively. The indicator function 1 {x=y} equals one if x = y, and zero if x = y. If two sets B ⊂ A, A\B denotes the set {x |x ∈ A, x / ∈ B}. If A is a finite set, then we let ∆A represent the set of probability distributions over A, i.e., ∆A := {p : A ← R + | a∈A p(a) = 1}. The main notation of this paper is introduced in Section 2.1. For sections that are dense with parameters, we include separate notation tables to help readers navigate.
The remainder of the paper is organized as follows. Section 2 first introduces a general schema of RL methods to help readers navigate in the rich universe of RL algorithms. After that, Section 2 gives an overview of literature that studies how RL is leveraged to fight against different types of attacks on cyber systems. While Section 2 focuses on the different attacks, Section 3 centers on three major classes of vulnerabilities: the posture-related vulnerabilities, the information-related vulnerabilities, and the human-related vulnerabilities. In Section 3, we discuss how each vulnerability can be addressed by RL. Following the discussion of 3, Section 4 delves into three application domains, each representing one class of vulnerabilities, and presents design methodologies for cyber resilience. Compared to previous sections that are about RL for cyber security, Section 5 is about security of RL, which introduces potential security threats faced by RL itself. In Section 5, we discuss the vulnerabilities of RL algorithms and provides an overarching review on this emerging research topic. We conclude the paper in Section 6 and discuss several directions for future research.

Reinforcement Learning and Its Applications in Cyber Resilience and Defense
The massive scale of Internet-connected or networked-connected systems expands the attack surface. The attacker can leverage one vulnerability in massive systems to penetrate the system and launch successful attacks. Most Internet-connected systems have dynamic nature due to their movement or changes in the environment. Hence, attacking windows can appear from time to time depending on the underlying dynamics, encouraging the attackers to look for opportunities persistently. The massiveness and the dynamics of the system require protecting and securing mechanisms to be responsive, adaptive, and scalable.
With the recent advances in Artificial Intelligence (AI), both the defense and the attacking sides have started to use AI techniques, especially machine learning (ML) methods, to obtain an edge. Sophisticated attackers leverage ML techniques to locate vulnerabilities, stay stealthy, and maximize the effect of attacks. Defenders employ ML techniques to adaptively prevent and minimize the impacts or damages. Both supervised and unsupervised learning methods have been used widely by the defenders for spam filtering [26], intrusion detection [27,28], zero-day vulnerability forecasting [29], and malware detection [30]. However, these traditional ML methods cannot provide dynamic and sequential responses against cyber attacks from the dark side with unknown patterns and constantly evolving behaviors.
As a branch of ML, Reinforcement Learning (RL) is a learning paradigm that learns from its own experience over time through exploring and exploiting the unknown and changing environment. RL provides suitable tools for defenders to take sequential actions optimally without or with limited prior knowledge of the environment and the attacker. RL approaches allow the defender to capture various types of dynamics (e.g., stochastic or deterministic), a wide range of protective and defensive actions ( high-dimensional continuous state space or discrete state space), and different kinds of system states. RL demonstrates an excellent fit for cyberspace where cyber attacks become increasingly advanced, swift, and omnipresent [31,32,33]. In recent years, the development of deep learning has fueled the successful synthesis of Deep Reinforcement Learning (DRL). The idea of DRL is to utilize a neural network to approximate complicated functions with high-dimensional inputs. The integration of deep learning enhances the conventional RL methods to capture the massive scale of many Internet-connected systems such as mobile networks and IoT systems [34,35].

RL Framework and Algorithms
The RL agent learns a near-optimal, if not optimal, to protect, prevent, or recover from attacks by constantly exploring and exploiting the environment in a feedback manner, as is shown in Fig.  2. In Fig. 2, s represents the security state of the cyber system, and the cost c is the monetary cost or performance degradation of the system induced by attacks and operations. The RL agent, or the defender, aims to find an action a that optimally modifies the system configuration or other parameters, leading to changes in the security state and the cost. The feedback loop forms an iterative process of agent-environment interactions.
RL is underpinned by a mathematical framework called Markov Decision Process (MDP). An MDP is denoted by a 5-tuple S, A, c, P, β , where S is the state space containing all possible states of the cyber system. The action space A denotes the actions available for the defender to protect the cyber system, to recover from the damage caused by attacks, or to mitigate the effect of attacks. The cost c depends on the current and/or the next security states, and the current action, which is usually denoted by a function that maps S × A × S → R + . The transition kernel P defines the rule of how the system state evolves based on the actions taken by the defender. The discount factor β is a weighting factor that assigns more weight to current costs than future costs. The goal of an MDP problem is to find a policy π t : S → A to minimize a certain form of the accumulative costs ∞ t=0 β t c t over time. MDPs are generic modeling tools that can model the dynamic and feedback nature of various types of cyber systems. Conventional MDP approaches require the knowledge of the transition probability of the cyber system and the explicit definition of the reward function to find an appropriate strategy. However, in reality, obtaining such information is prohibitive for two reasons. The first is due to the complexity of the cyber system and its dynamics. The second is because the attackers are usually on the dark side whose behavior is unknown to the defender. RL solves the MDP problem without the knowledge of the transition kernel P and the cost function c. Instead of utilizing a prior known transition kernel and cost function, RL agent learns the optimal policy from sequences of states {s t } t∈Z , costs {c t } t∈Z , and actions {a t } t∈Z , which are obtained by interacting with the environment over time.
Having been actively studied for decades, RL has a rich universe of algorithms that help the agent find a satisfactory policy. Covering every single algorithm is beyond the scope of this paper. Here, we highlight a general schema of RL models shown in Fig. 3 and present a concise taxonomy of RL algorithms.

Model-Based and Model-Free
One advantage of RL over classic MDP and control is its ability to cope with the unknown environment (e.g., cost function c, dynamics P). Its ability to tackle unknown environments makes RL an appealing tool for cyber resilience. There are two types of methods that tackle the curse of modeling: the model-based RL and the model-free RL. Here, a 'model' means an ensemble of acquired environmental knowledge. If the model (the cost function c and the transition kernel P) is known, methods in classic MDP (e.g., value iteration, policy iteration, linear programming, etc.) can be applied directly [36].
In model-based RL, the agent estimates the cost function as well as the transition kernel using data sequences {s t , c t , a t , s t+1 } t∈Z . Once elements of the model are constructed/estimated from samples, the agent can apply classic methods in MDP directly to obtain a near-optimal policy. There are occasions where the model is explicitly given and can be accessed directly by the agent. For example, the MCTS (AlphaGo/AlphaZero) algorithm relies on a fully known environment because the rules of the GO game are exactly known [37]. Nevertheless, in most cases, the model is unknown due to the complexity or opaqueness of the environment. The agent estimates the elements in the environment first, then applies classic MDP methods. Typical examples of this kind includes the MBMF algorithm [38], the World Models algorithm [39], the I2A algorithm [40], and etc.
The model-free RL does not predict the environment parameters but directly seeks the optimal policy using the samples. The idea is to look constantly for the policy that produces higher rewards. An example is the Q-learning algorithm where the agent chooses, with a high probability, the action corresponding to the highest Q value. The Q-algorithm converges to an optimal Q-value function under proper conditions, which produces an optimal policy. The difference between model-based and model-free RL is whether the agent needs to construct/estimate the model of the environment (e.g., the transition kernel and the cost function). In Section 5.1, we will discuss the vulnerabilities of model-based RL and model-free RL under adversarial attacks on the cost samples.

Value-Based and Policy-Based
The model-free RL has two main categories for optimizing the policy: value-based methods and policy-based methods. The value-based methods usually employ a value-action function Q : S × A → R+, which is also referred to as the Q function. The most population algorithm of the value-based method is the Q-learning algorithm, whose goal is to find the optimal Q-values that satisfy the Bellman equation [36] Q(s, a) = c(i, a) + β s p(s, s , a) min a Q(s , a ), for i ∈ S, a ∈ A, where p(s, s , a) is the probability that the state of the cyber system at the next step is s given the current state s and the current action a. Without the knowledge of the transition probability p(·, ·, ·) and the cost function c(·, ·), the RL agent can update its Q-values by interacting with the environment: where the sequences of states {s t } t∈Z , costs {c t } t∈Z are from the environment and the sequence of actions {a t } t∈Z are chosen by the agent. The policy-based RL focuses on the policy directly. The policy-based method updates the policy iteratively to minimize the accumulative costs over a period of time without explicitly constructing a value function. The adoption of policy-based RL allows the agent to parameterize the policy, which often results in better convergence and is suitable for continuous or high dimensional action space. Frequently used policy-based algorithms include the Policy Gradient (PG) algorithm proposed by Sutton et al. [41], the Trust Region Policy Optimization (TRPO) algorithm [42], etc.
Combining the value-based and the policy-based methods gives rise to the actor-critic class of algorithms. The actor-critic method leverages the value-based methods to build a value function to improve sample efficiency and relies on the policy-based methods to update the policy parameterization. Common actor-critic RL algorithms include the Actor-Critic (AC) algorithm [43] and its variants: the Asynchronous Advantage Actor-Critic (A3C) algorithm [44], the Deterministic Policy Gradient (DPG) algorithm [45], etc.

Function Approximation
With the curse of modeling being tamed, the next is to discipline the curse of dimensionality. The curse of dimensionality appears when the state space or the action space is continuous or prohibitively large, which is the situation we often encounter in cyber systems. Function approximation is a technique that addresses the curse of dimensionality. A function approximator is selected to approximate the value function or the policy function so that the learning algorithms only have to deal with a few parameters. Common function approximators are linear function approximator [36,43,46], multiscale approximator [47], neural networks [33,44,48,43], etc. Recent development in Deep Neural Networks (DNNs) in the last decade enables the application of Deep Reinforcement Learning (DRL). In DRL, the function approximator is a neural network with multiple layers between the input and output layers. A wide variety of DNNs, such as convolutional neural networks, recurrent neural networks [48] and transformers [49], has been developed to handle different application scenarios. One can refer to Section 7 of [48] and Section 9 of [43] for a detailed discussion on the selection of function approximators.

Other RL Settings
In the training phase, the RL agent chooses a policy that interacts with the environment to generate more samples and constantly improve the policy using the generated samples. The RL agent accumulates knowledge about the cyber system. The agent has to make a trade-off between learning more about the systems or pursuing what seems to be the most promising defense strategy with the samples gathered so far. This is called the exploration-exploitation trade-off in choosing how to sample from the environment (or the cyber systems in our case). Commonly used sampling strategies include the greed strategy (i.e., choose the action currently unknown as the best), the -greedy exploration (i.e., applies the greedy action with probability 1 − and otherwise selects an action uniformly at random), or the Thompson sampling [50].
In many RL applications, the agent is equipped with a replay memory that stores the past experience/samples of the agent. The storage of past experience allows the agent to reuse the samples later and process samples in batches to achieve a reasonably stable data distribution [48]. With replay memory, the agent can choose either on-policy or off-policy methods. On-policy methods evaluate and improve the policy using the samples generated by the same policy. In off-policy methods, learning is done using samples in replay memory that are not necessarily generated under the current policy but different policies. The off-policy methods allow the defender to leverage data from historical security events to learn a defense policy. A typical example of the on-policy method is SARSA [43] where the agent executes an action from a -greedy policy and uses the sample generated by the action to update the current policy: (2) Q-learning algorithm in (1)) is a frequently used off-policy method. Even though it adopts the -greedy policy to generate actions, the update of Q-values follows the greedy policy as is indicated by the min operator in (1). With basic knowledge of RL methods, in the following section, we discuss the literature about how RL can be used to tackle different attacks in cyber systems.

Review of RL Against Cyber Attacks
The rapid development of Information and Communication Technologies (ICTs) has fueled significant growth in Internet of Things (IoT) usage and Cyber-Physical Systems (CPS) deployment. CPS involves embedded computers and networks used to monitor and control the physical processes, with feedback loops where physical processes affect computations and vice versa. The Internet of Things (IoT) refers to a network comprised of physical objects capable of gathering and sharing electronic information, with physical objects receiving orders from the network and the network receiving information from the physical objects. Both IoT and CPS involve computation units, networking units, and physical entities that form a feedback cycle with human or machine decisions in the loop. Due to the wide adoption of IoT and CPS in many domains such as manufacturing, power, agriculture, battlefield, homes, etc., security threats faced by IoT and CPS systems need to be well addressed.
First, in a CPS, traditional physical controllers have been replaced by micro-computers and processors with embedded operating systems [51]. Micro-computers provide advantages to the CPS, such as flexible configuration with an accessible human-machine interface and digital communication abilities that enable remote control and access. However, the embedded software opens up doors for software attacks such as code injection attacks, cross-site scripting attacks when using a web server for system configuration, and malware attacks. Second, CPS is networked, meaning that the essential components in the physical systems are often connected to corporate networks and the Internet. Security issues may be raised because complete isolation is difficult to achieve. Connectivity can occur in unexpected ways. For example, Stuxnet, a malicious computer worm accused of stymieing Iran's nuclear-fuel enrichment program, was introduced to the target environment via an infected USB flash drive [52]. The security challenges of CPS will become more severe as the scale and scope of the Internet and other communication techniques grow.
Similarly, IoT is also facing security challenges amid the increasing adoption of IoT technology. IoT employs advanced computing and communication technology such as Zigbee, Bluetooth, radiofrequency identifications (RFIDs), and cloud computing. The employment of these technologies provide opportunities for the attacker to launch cyber attacks, such as DoS attack, eavesdropping, data injection attacks, etc. Beyond that, unlike CPS, IoT usually encompasses small physical units with lower power that are mobile and geographically distributed. Hence, IoT is more susceptible to physical attacks. Due to the dynamic and feedback nature of CPS and IoT, RL has been applied to address the security and privacy challenges faced by these systems. In this section, we provide a brief review of the applications of RL for the security of CPS and IoT. The review follows a taxonomy based on the attack types and different RL algorithms.

RL Aganist DoS attacks
DoS is the most common attack among all the attacks targeting networked systems such as IoT and CPS [53,54]. A denial-of-service (DoS) attack is a tactic used by attackers for overloading a machine or network to make it unavailable. DoS attacks on networked systems bring serious threats to people and induce direct or indirect financial losses. Since many CPS relies heavily on real-time communication, the sudden unavailability of information may render serious issues. For example, suppose a critical chemical reaction process is unstable in open-loop control. In this case, a continuous DoS attack on the actuator may introduce irreversible damage to the system and the beings around it. Liu et al. [55] have studied the resilient control problem for CPS under DoS attacks. The DoS attacks target the measurement and control channels to jeopardize the closed-loop system's functionality. The authors designed a resilient control strategy that can achieve a certain degree of stability and performance for the CPS even when under DoS attacks. To compute such a resilient control strategy, the authors have employed an on-policy RL method called SARSA to calculate the strategy. Dai et al. [56] have studied the remote state estimation of CPS with multiple sensors. The sensors transmit the message through several communication channels subject to possible DoS attacks. The authors leverage Q-learning to compute the communication strategy, i.e., to decide which communication channel to use.
Due to its distributed nature with limited energy, IoT is often targeted by Distributed DoS (DDoS) attacks. Distributed DoS often results in malfunction in an enormous number of IoT devices within a short time window. Xu et al. [22] have proposed a DDoS detection scheme based on Hidden Markov Models and cooperative RL. The authors leverage DQN to compute optimized strategies of information exchange among the distributed multiple detectors to ensure a decent detection accuracy without much load on information communications among the detectors. Later in 2013, Malialis et al. [21] have proposed a protective mechanism to protect IoTs from DDoS attacks by router throttling. A multi-agent RL framework is applied where the agents representing the routers learn to throttle traffic toward a victim server. However, this method will suffer scalability issues as the scale of IoT increases. To tackle this problem, the same authors proposed a coordinated design on top of the previous RL-based router throttling method [57]. More recently, Liu et al. have proposed a DQN based framework to mitigate the effect of the DDoS attack, which can increase or slow down the attack traffic flow [58]. Most papers that leverage RL to mitigate the effect of DoS attacks are for computational purposes when the behavior model of the attacker is unknown to the agent.

RL Aganist Spoofing Attacks
A spoofing attack happens when a malicious party impersonates another device/user on a network to launch other attacks, steal data, spread malware, or bypass access control [59]. There are many things the attackers can forge to make their attacks pan out: An IP address, a phone number, GPS location, etc. Out of all the nefarious scenarios, the following few become increasingly impactful for cyber security nowadays. ARP (Address Resolution Protocol) spoofing is a spoofing attack in which a malicious attacker sends falsified ARP messages over a local area network. This manipulation can make all traffic redirected to the attacker's device before it can reach its legitimate destination. Another common type of spoofing attack is the IP spoofing attack, in which the attacker impersonates the IP address and obfuscates the actual online identity of the packet sender. The penetration using spoofing attacks gives wings to the attacker to launch further attacks such as man-in-the-middle or DoS attacks.
Xiao et al. have formed a zero-sum authentication game to capture the interplay between the legal receiver and spoofers [60,61]. The authors leverage Q-learning algorithms to compute the spoofing detection strategy. The receiver uses this strategy to choose the test threshold in the hypothesis test of the spoofing detection to maximize its expected utility based on Bayesian risk to detect the spoofer. Beyond the passive defensive method such as detection, Elnaggar et al. [62] have proposed an inverse RL approach that leverages historical measurement and control data to forecast the objective of spoofers. The authors studied an autonomous vehicle equipped with multiple sensors and tasked to perform go-to-goal navigation. The spoofer aims to hijack the vehicle to an adversary-desired location while staying stealthy. The agent can prevent the vehicle from reaching the adversary-desired location by leveraging the IRL method.

RL Aganist False Data Injection Attacks
False data injection (FDI) attacks affect the data integrity of packets by modifying their payloads, which are in general more difficult to detect but have less been investigated [63]. Li et al. [64] proposed a DQN-based RL method, on behalf of the attacker, to jeopardize the data analysis results by optimally injecting fake data conflicting with ground truth in crowdsensing systems. Similarly, In [65], Chen et al. come up with a novel strategy of false data injection (FDI) attacks on behalf of the attacker, aiming to distort the regular operation of a power system by automatic voltage controls. The problem of finding an optimal attack strategy is formed as a partially observable MDP, and a Q learning algorithm is proposed to enable online learning and attacking.
Instead of designing optimal FDI attacking strategies for the attacker, Kurt et al. [66] designed a defense strategy against a series of cyber attacks in smart grids. The authors proposed a modelfree RL algorithm to detect cyber attacks, including FDI attacks, on the fly without knowing the underlying attack model. Numerical results show that the proposed detection system can detect FDI attacks with a precision of 0.9977 and a recall of 1. This section has introduced different types of attacks, how they can cause damage to the systems, and how RL can mitigate the damage. The following section focuses on the three major classes of vulnerabilities in cyber systems and discusses how RL can help address each class of vulnerabilities.

Resilient Cyber Defense Against Vulnerabilities
As discussed in Section 1, the preparation and prevention mechanisms for perfect security are too costly to implement. Response and recovery mechanisms are necessary to mitigate successful attacks and harden the system security. RL is an efficient tool to achieve this goal through online learning, correction, and adaptation. In this section, we provide a literature overview on cyber resilience and focus on methodologies that rely on feedback mechanisms, which include RL and control-theoretic approaches. We refer readers to several recent reviews on game theory for cybersecurity [67] and cyber deception [68], and Deep Reinforcement Learning (DRL) for cybersecurity [33]. The design of CRMs depends on the type of vulnerabilities that the cyber system aims to mitigate. We group the common vulnerabilities into three categories, namely, (1) the posture-related vulnerabilities, (2) the information-related vulnerabilities, and (3) the human-related vulnerabilities.
The posture-related vulnerabilities refer to a defender's naturally disadvantageous security posture in comparison to the attacker's; e.g., the defender has to be constantly vigilant and legitimately defend the entire attacker surface, while the attacker only needs to succeed once at a single location. Due to the disadvantage in security posture, the defender with limited resources cannot afford to prepare for all possible attacks. The information-related vulnerabilities refer to the defender's information disadvantage, especially when facing deceptive and stealthy attacks. The defender cannot make a meticulous plan to protect his assets if he does not have the capability of mapping out the attack paths or predicting the targets. The human-related vulnerabilities are the results of human misbehavior and cognition limitation. The vulnerabilities of all human groups in the cyber system can expose the system to cyber threats and undermine cyber resilience. Human users and insiders can unintentionally fall victim to phishing attacks or intentionally break security rules for their convenience. Human operators and network administrators in charge of real-time monitoring and inspections of alerts and system status can suffer from alert fatigue. Human vulnerabilities are often exploited by attackers as the first step who aim to penetrate a computing system. Preparation and prevention mechanisms, e.g., security training and regulations, are not sufficient to protect them from attacks.
We briefly review the existing technologies to mitigate the three types of vulnerabilities in Section 3.1, 3.2, and 3.3, respectively. In particular, we emphasize the RL-based CRMs and introduce three cyber resilience designs, i.e., MTD in Section 4.1, honeypots in Section 4.2, and human-assistive technologies in Section 4.3, as the representative examples of the RL solutions to these three types of vulnerabilities, respectively.

Posture-Related Vulnerability
The technologies to mitigate posture-related vulnerabilities have been extensively studied in the literature. Moving Target Defense (MTD) is one of the modern technologies to neutralize attacker's position advantage by creating reconnaissance difficulties and uncertainties for attackers. There is a surge of recent literature on using RL to choose an adaptive configuration strategy to maximize the impact of MTD with particular focuses on the dynamic environment [69], reduced resource consumption [70], usability [11], partially observable environment [71,72], and multiagent scenarios that contains both the characteristics of the system and the adversary's observed activities [73,72].
Other techniques have been proposed to protect disadvantageous cyber systems from falling victim to resourceful and determined attacks such as Distributed Denial-of-Service (DDoS) or powerful jamming attacks. For example, [58] mitigates DDoS Flooding in Software-Defined Networks; [74] introduces a multi-objective reward function to guide an RL agent to learn the most suitable action in mitigating application-layer DDoS attacks; [75] proposes a feature adaption RL approach based on the space-time flow regularities in IoV for DDoS mitigation. Among the works of RL for DDoS attack, there has been recent attention on large-scale solution [21,22] via cooperative RL, low-rate attacks in the edge environment [76], and sparse constraint in cloud computing [77]. Besides DDoS attacks, RL has a wide application in mobile edge caching [78], mobile offloading for cloud-based malware detection [79], and host-based and network-based intrusion detection [80,81].

Information-Related Vulnerability
Deception is a ubiquitous phenomenon and has been used as a proactive defense method. Honeypots and the related cyber deception technologies, e.g., honeyfiles, honeynet, honeybot, etc., have been widely used to mitigate stealthy and deceptive attacks. In particular, RL has been widely used to develop self-adaptive honeypots in normal conditions [82,83,84] and resource-constrained environment [85]. As attackers become more sophisticated, they manage to fingerprint and evade honeypots [86]. This reduces the size of a captured dataset and the chance to extract threat intelligence from it. Thus, there is an urgent need for stealthy honeypots [87] that can counter fingerprinting. In [88], the authors use RL to conceal the honeypot functionality. Besides honeypot fingerprinting, RL is also used to extract the most threat intelligence [89,90,23].
Besides honeypots, there are emerging technologies to detect and respond against deceptive attacks. In [91,92,93], the authors model the interactions between stealthy attackers and the defenders as a dynamic Bayesian game of discrete or continuous type where multi-stage response strategies are designed based on belief updates. The convergence of continuous-type Bayesian game under a sequence of finer discretization schemes is discussed in [94]. In [95], the authors design the information revealed to the users and attackers to elicit behaviors in favor of the defender. In [96] and [97], RL is used to optimally deploy the deception resources and emulate adversaries, respectively. Many works have attempted to address various spoofing attacks using RL. In [61], the authors propose RL-based spoofing detection schemes where the interactions between a legitimate receiver and spoofers are formulated as a zero-sum authentication game. In [98], the authors model the behavior of exploring face-spoofing-related information from image sub-patches by leveraging deep RL. Developing counter-deceptive and defensive deceptive technologies is in its infancy. RL is a promising tool to make the design quantitative, automatic, and adaptive.

Human-Related Vulnerability
We classify human vulnerabilities into acquired and innate vulnerabilities, depending on whether they can be mitigated through short-term security training and awareness programs. Social engineering [99] is a common attack vector that targets acquired human vulnerabilities such as fear to express anger, lack of assertiveness to say no, and the desire to please others. Threat actors use psychological manipulation techniques to mislead people to break normal security procedures or divulge confidential information. The authors in [100] have used RL to detect cyberbullying automatically, and [101,102] have proposed a feedback learning framework, e.g., RL, to fight against social engineering attacks. As a representative form of social engineering, phishing attacks use email or malicious websites to serve malware or steal credentials by masquerading as a legitimate entity. Non-technical anti-phishing solutions include security training and education programs, while technical solutions include blacklisting, whitelisting, and feature-based detection. To handling zero-day phishing attacks, (deep) RL has been used both to detect phishing emails [103], phishing websites [104], spear phishing [105], and social bots [106] in Online Social Networks (OSN). Attackers can also exploit human innate vulnerabilities induced by bounded attention and rationality. The authors in [24,25] have identified a new type of attacks called Informational Denial-of-Service (IDoS) attacks that can exacerbate the human operators' cognition overload by generating a large number of feints and hiding real attacks among them.
Most of the existing works take humans as an independent component in the cyber system and aim to compensate indirectly for the human vulnerability through additional mechanisms. An alternative way is to directly affect the human component and consider an integrated human-cyber system. Due to the unpredictability and modeling challenges of human behaviors, RL and feedback control serves as the tool to determine how to affect human incentives and perceptions effectively and efficiently. In [107], the penalty and reward are changed adaptively through a feedback system to improve compliance of human employees and mitigate insider threats. In [108], RL is used to develop the optimal visual aids to enhance users' attention and help them identify phishing attacks. In [25], the authors use RL to determine resilient and adaptive strategies for alert and attention management. Developing security-assistive technologies that directly affect humans is in its infancy, and it is a promising direction to explore further.

Cyber-Resilient Mechanism Designs and Applications
The system vulnerabilities and attacker's information and resource advantages have made the response and recovery mechanisms a necessity. To achieve cyber resilience, the defender needs to make the response and recovery mechanisms configurable, adaptive, and autonomous. RL and feedback control provide a rich set of tools to achieve these three features.
In this section, we introduce the following three cyber resilience designs, i.e., MTD strategies for security-usability trade-off [11] in Section 4.1, honeypot strategies for attack engagement [23] in Section 4.2, and human-assistive alert management strategies [24,25] in Section 4.3, as the representative examples of RL solutions to three vulnerabilities in Sections 3.1, 3.2, and 3.3, respectively. For each application, we first introduce the necessary background, motivation, and challenges of the defense technologies. Then, we introduce the feedback and learning models to capture the online behaviors of the players, e.g., the defender, the attackers, and the users. Finally, we illustrate how RL can enable cyber resilience.
The CRMs in all three works guarantee asymptotic convergence, but the learning speed varies based on the complexity of the objective strategy and the learning environment. Fast convergence can reduce the response time in Fig. 1 and lead to a more resilient solution to enhance cyber security performance. Many works in the RL literature, including approximations [109], transfer learning [110], and meta learning [111], have improved the learning speed and reduced sample complexity. However, few works have focused on applying the above success to design CRM. We envision seeing more works that apply those RL algorithms to the cyber domain and thus enable efficient learning and timely responses. Besides developing general efficient RL algorithms, we can also tailor and improve the existing RL solutions to the characteristics of cyber systems. For example, it is sufficient to consider epsilon-optimality when the cyber system is not stationary; e.g., its state changes dynamically and stochastically based on the behaviors of users, defenders, and attacks.

Adaptive MTD Strategies for Security-Usability trade-off
Many legacy Information Technology (IT) systems operate in a relatively static configuration. As many configuration parameters, e.g., IP addresses, remain the same over long periods, advanced attackers can reconnoiter and learn the configuration to launch targeted attacks. MTD is a countermeasure that dynamically configures the system settings and shifts the attack surface to increase uncertainties and learning costs for the attackers.
Many networked IT systems nowadays consist of hierarchical layers and adopt the Defense-in-Depth (DiD) approach where a series of defensive mechanisms reside on each of these layers. Let N := {1, 2, · · · , N } denote the set of N layers in a system and V l := {v l,1 , v l,2 , · · · , v l,n l } be the set of n l system vulnerabilities that an attacker can exploit to compromise the system at layer l ∈ N . Assuming that perfect security is unattainable and attacks can penetrate the layered defense of different depths, MTD can be applied for all layers to interact and slow down the attacker's rate of penetration. At each layer l ∈ N , the defender can choose to change the system configuration from a finite set of m l feasible configurations C l := {c l,1 , c l,2 , · · · , c l,m l }. Different configurations result in different subsets of vulnerabilities among V l , which are characterized by the configurationvulnerability map π l : C l → 2 V l . Under the h-th configuration c l,h ∈ C l , the outcome of the configuration-vulnerability mapping, i.e., π l (c l,h ), represents the attack surface at layer l ∈ N .
After the attacker has reached the layer l ∈ N , the attacker can launch an attack a l,k from a finite set A l := {a l,1 , a l,2 , · · · , a l,n l }. Let γ l : V l → A l be the vulnerability-attack map that associates vulnerability v l,j ∈ V l with attack a l,k ∈ A l . Without loss of generality, there is a one-to-one correspondence between A l and V l , then the corresponding inverse map is denoted as γ −1 l : A l → V l . Attack action a l,k ∈ A l incurs a bounded cost D hk ∈ R + when the current attack surface π l (c l,h ) under configuration c l,h ∈ C l contains the vulnerability v l,j = γ −1 l (a l,k ). Otherwise, the attacker fails to exploit the existing vulnerabilities under the configuration and incurs zero damage. Thus, the damage caused by the attacker at layer l ∈ N , denoted by r l : A l × C l → R + , takes the following form: Indices & Sets: N, N Number and set of layers n l , V l Number and set of vulnerabilities at layer l: {v l,1 , v l,2 , · · · , v l,n l } C l Set of system configurations at layer l: {c l,1 , c l,2 , · · · , c l,m l } A l Set of attacks available at layer l: {a l,1 , a l,2 , · · · , a l,n l } Decision & Strategy: a l,k Attack targeting at vulnerability k at layer l ∆C l , ∆A l Distributions of set C l and A l f l = {f l,1 , f l,2 , · · · , f l,m l } ∈ ∆C l Defender's randomized strategy over C l g l = {g l,1 , g l,2 , · · · , g l,n l } ∈ ∆A l Attacker's randomized attacking strategy over A l c l,t (a l,t ) Action chosen by defender (attacker) at time t Other Variables: Cost induced by the attack indexed by k under h-th configuration r l : A l × C l → R + Damage caused by the attacker at layer l r S l,t (r A l,t ) Defender's (attacker's) estimate of the average risk µ The attacker's goal is to penetrate and compromise the system, while the defender aims to minimize the damage or risk. Since vulnerabilities are inevitable in modern IT systems, the defender adopts MTD to randomize between configurations and make it difficult for the attacker to learn and exploit the vulnerabilities at each layer. Fig. 4 provides a paradigmatic example to illustrate the benefit of MTD. At the first layer highlighted by the blue box, there are three vulnerabilities. Configuration c 1,1 in Fig. 4a has an attack surface π 1 (c 1,1 ) = {v 1,1 , v 1,2 } while configuration c 1,2 in Fig. 4b has an attack surface π 1 (c 1,2 ) = {v 1,2 , v 1,3 }. For each attack surface, the existing and nonexisting vulnerabilities are denoted by the solid and dashed arcs, respectively. Then, if the attacker takes action a 1,1 ∈ A 1 that exploits vulnerability v 1,1 but the defender changes the configuration from c 1,1 to c 1,2 , then the attack is thwarted at the first layer.
The authors in [11] model the conflict between a multi-stage attacker and a defender adopting MTD as a zero-sum game. At each layer l ∈ N , the defender's randomized strategy over the configuration set C l is denoted as f l := {f l,1 , f l,2 , · · · , f l,m l } ∈ ∆C l where ∆C l is the distribution of set C l . The attacker can also choose a randomized strategy g l := {g l,1 , g l,2 , · · · , g l,n l } ∈ ∆A l over the set of feasible attacks at layer l ∈ N to increase his chance of a successful vulnerability   compromise. In particular, f l,h ∈ [0, 1] and g l,k ∈ [0, 1] represent the probabilities of the defender taking configuration c l,h and the attacker taking action a l,k , respectively, at layer l ∈ N . Under the mixed-strategy pair (f l ∈ ∆C l , g l ∈ ∆A l ), the defender's expected cost r l is given by The defender's optimal randomized strategy f * l ∈ ∆C l against the worst-case attacks can be obtained from the mixed strategy saddle-point equilibrium (SPE) (f * l ∈ ∆C l , g * l ∈ ∆A l ) of the zero-sum game where the game value r(f * l , g * l ) is unique, i.e.,

Optimal Configuration Policy via Reinforcement Learning
In practice, the costs D hk , ∀h ∈ m l , k ∈ n l , are unknown and subject to noise. Thus, the attacker and the defender need to learn the cost r l independently during their interaction over time, which is referred to as the procedure of 'sense' in Fig. 5. Then, each player updates his policy based on the estimated cost and then takes an action based on the updated policy, which are referred to as the procedures of 'learn' and 'act', respectively, in Fig. 5.
The subscript t denotes the strategy or cost at time t. Due to the non-cooperative environment, the attacker does not know the configuration and the defender does not know the attack action throughout their interaction. Thus, at time t, they independently choose actions c l,t ∈ C l and a l,t ∈ A l according to strategies f l,t and g l,t , respectively. Then, they commonly observe the cost r l,t as an outcome of their action pair (c l,t , a l,t ). Based on the observed cost at time t, the defender and the attacker can estimate the average riskr S l,t : C l → R + andr A l,t : A l → R + , respectively, as follows:r S l,t+1 (c l,h ) =r S l,t (c l,h ) + µ S t 1 {c l,t =c l,h } (r l,t −r S l,t (c l,h )), ∀h ∈ {1, · · · , m l }, r A l,t+1 (a l,k ) =r A l,t (a l,k ) + µ A t 1 {a l,t =a l,k } (r l,t −r A l,t (a l,k )), ∀k ∈ {1, · · · , n l }, where µ S t and µ A t are the payoff learning rates for the defender and the attacker, respectively. The indicators in (6) mean that the defender and the attacker only update the estimate average risk of the observed action c l,t and a l,t at the current time t, respectively.
The defender uses the estimated riskr S l,t+1 (c l,h ) to update his configuration strategy from f l,t to f l,t+1 . The strategy change involves a reconfigure cost to maneuvering the defense resources and altering the attack surface from π l (c l,t ) to π l (c l,t+1 ), where c l,t and c l,t+1 are selected according to f l,t and f l,t+1 , respectively. Define the reconfigure cost as the relative entropy between two consecutive strategies: Then, the reconfigure cost R S l,t is added to the original expected cost m l h=1 f l,h,t+1r S l,t (c l,h ) with a parameter S l,t > 0 to quantify the trade-off between the security and the usability. Thus, the defender aims to solve the following optimization problem (SP) at time t: which has the following closed-form solution in (9) and the optimal value W S l,t in (10).
When the value of S l,t is high, the configuration policy changes less and is more usable. However, it is also easier for the attacker to learn the policy and thus reduce the security. If S l,t → ∞, then the configuration policy remains the same f l,h,t+1 = f l,h,t and the optimal value in (10) equals − m l h=1 f l,h,tr S l,t (c l,h ). If S l,t → 0, then the optimal value in (10) equals min c l,h ∈C lr l,t (c l,h ). Analogously, it takes an attacker time and energy to change the attack strategy to explore new vulnerabilities and exploit them, which leads to the following optimization problem (AP)) for the attacker. g l,k,t+1 ln g l,k,t+1 g l,k,t , which has the following closed-form solution in (12) and the optimal value W S l,t in (13).
The parameter A l,t > 0 achieves the trade-off between the attacker's benefit of attacks and the cost to learn the most effective attack. With all these notations introduced, we use Fig. 5 to illustrate the attacker's (resp. the defender's) risk learning, policy update, and action implementation in the light orange (resp. light blue) background. In either the attacker's adversarial learning or the defender's defensive learning, the learning rule does not depend on the other player's action, yet the observed payoff depends on both players' actions.  Figure 5: Adaptive learning of the multistage MTD game at layer l where the adversarial and defensive learning is presented in red and green, respectively. In both players' learning feedback, the procedures of 'sense', 'learn', and 'act' correspond to the boxes of 'information acquisition', 'decision making', and 'security configuration' in Fig. 2, respectively.
The dynamics for the mixed strategy update (represented by the procedure of 'learn' in Fig. 2) in (9) and (12) can be generalized into the following two learning dynamics (14) and (15), respectively, with learning rates λ S l,t , λ A l,t ∈ [0, 1].
If the learning rates λ S l,t = 1 and λ A l,t = 1, the learning dynamics (14) and (15) are the same as (9) and (12), respectively. According to the stochastic approximation theory, the learning dynamics (14) and (15) converge to the SPE of the game in (5) under mild conditions [11].

Adaptive Honeypot Configuration for Attacker Engagement
Traditional cybersecurity techniques such as the firewall and intrusion detection systems rely on low-level Indicators of Compromise (IoCs), e.g., hash values, IP addresses, and domain names, to detect attacks. However, advanced attacks can revise these low-level IoCs and evade detection. Thus, there is an urgent need to disclose high-level IoCs, also referred to as threat intelligence, such as attack tools and Tactics, Techniques, and Procedures (TTP) of the attacker. As a promising active cyber defense mechanism, honeypots can gather essential threat intelligence by luring attackers to conduct adversarial behaviors in a controlled and monitored environment.

Variables
Meaning Index for time and stage s k ∈ S State at stage k a k ∈ A(s k ) Engagement action tr,z Transition kernel and sojourn distribution r 1 , r 2 , r Transition reward, sojourn reward, and investigation reward r γ Equivalent investigation reward under discounted factor γ ∈ [0, ∞) π ∈ Π := S → A Engagement strategy u(s 0 , π), v(s 0 ) Long-term expected utility and value function starting from s 0 ∈ S A honeynet is a network of honeypots, which emulates the real production system but has no production activities nor authorized services. Thus, an interaction with a honeynet, e.g., unauthorized inbound connections to any honeypot, directly reveals malicious activities. From an attacker's viewpoint, the production network and the honeypot network share the same structure as shown in Fig. 6

Honeynet
Production Systems Figure 6: The honeynet in red emulates and shares the same structure as the targeted production system in green.
An attacker from the production system or external network can be attracted to the honeynet. Then, the attacker can move inside the honeynet through either physical (if the connecting nodes are real facilities such as computers) or logical (if the nodes represent integrated systems) links.
The attacker's transition is restricted by the network topology. Once the attack launches an attack at a honeypot node, the defender gains an investigation reward by analyzing the attack behaviors and extracting high-level IoCs. Since the defender can obtain more threat intelligence when the attack sustains for a longer time, the investigation reward increases with the engaging time. The defender aims to dynamically configure the honeynet through proper engagement actions to obtain more threat intelligence while minimize the following three types of risks. T1: Attackers identify the honeynet and thus either terminate on their own or behave misleadingly in honeypots.
T2: Attackers circumvent the honeywall and use the honeypots as a pivot to penetrate other production systems.
T3: The cost to engage attackers in the honeypots outweighs the resulted investigation reward.
To strike a balance between maximizing the investigation reward resulted and minimizing the risks, the authors in [23] model the attacker's stochastic transition in the honeynet as an infinite-horizon Semi-Markov Decision Process (SMDP) to quantify the long-term reward and risks. The SMDP consists of the tuple {t ∈ [0, ∞), S, A(s j ), tr(s l |s j , a j ), z(·|s j , a j , s l ), r γ (s j , a j , s l ), γ ∈ [0, ∞)}. We illustrate each element of the tuple through a 13-state example in Fig. 7. Each node in Fig. 7 represents a state s i ∈ S, i ∈ {1, 2, · · · , 13}. At time t ∈ [0, ∞), the attacker is either at one of the honeypot nodes denoted by state s i ∈ S, i ∈ {1, 2, · · · , 11}, at the normal zone s 12 , or at a virtual absorbing state s 13 once attackers are ejected or terminate on their own. At each state s i ∈ S, the defender can choose an action a i ∈ A(s i ). The action set A(s i ) is finite and depends on the state s i ∈ S. For example, at honeypot nodes, the defender can conduct action a E to eject the attacker, action a P to purely record the attacker's activities, low-interactive action a L , or high-interactive action a H , i.e., A(s i ) := {a E , a P , a L , a H }, i ∈ {1, · · · , 11}. At normal zone, the defender can choose either action a E to eject the attacker immediately, or action a A to attract the attacker to the honeynet by generating more deceptive inbound and outbound traffics in the honeynet, i.e., A(s 12 ) := {a E , a A }. No actions needed in the virtual absorbing state, i.e., A(s 13 ) := ∅.
Based on the attacker's current state s j ∈ S and the defender's action a j ∈ A(s j ), the attacker transits to state s l ∈ S with probability tr(s l |s j , a j ) and the sojourn time at state s j is a continuous random variable with probability density z(·|s j , a j , s l ). Once the attacker arrives at a new honeypot n i , the defender dynamically applies an interaction action at honeypot n i from A(s i ) and keeps interacting with the attacker until the attacker's next transition. Since the defender makes decision at the time of transition, the above continuous-time model over t ∈ [0, ∞) can be transformed into a discrete decision model at decision epoch k ∈ {0, 1, · · · , ∞}. The time of the attacker's k th transition is denoted by a random variable T k , the landing state is denoted as s k ∈ S, and the adopted action after arriving at s k is denoted as a k ∈ A(s k ).

Optimal Engagement Strategy via Reinforcement Learning
In practice, the defender cannot know the exact SMDP model, i.e., the investigation reward, the attacker's transition probability (and even the network topology), and the sojourn distribution. Thus, the defender learns the optimal engagement policy iteratively during the honeynet interaction with the attacker through the Q-learning algorithm for SMDP [112], i.e.,  where s k is the current state sample, a k is the current selected action, α k (s k , a k ) ∈ (0, 1) is the learning rate,s k+1 is the observed state at next stage,r 1 ,r 2 is the observed investigation rewards, andτ k is the observed sojourn time at state s k . When the learning rate satisfies ∞ k=0 α k (s k , a k ) = ∞, ∞ k=0 (α k (s k , a k )) 2 < ∞, ∀s k ∈ S, ∀a k ∈ A(s k ), and all state-action pairs are explored infinitely, max a ∈A(s k ) Q k (s k , a ), k → ∞, in (18) converges to value v(s k ) with probability 1. At each decision epoch k ∈ {0, 1, · · · }, the action a k is chosen according to the -greedy policy, i.e., the defender chooses the optimal action arg max a ∈A(s k ) Q k (s k , a ) with a probability 1 − , and a random action with a probability . Fig. 8 illustrates an exemplary learning trajectory of the state transition and sojourn time under the -greedy exploration policy, where the chosen actions a E , a P , a L , a H are denoted in red, blue, purple, and green, respectively.
We plot the entire feedback learning structure in Fig. 9. The environmental uncertainties represented by the red background result from the attacker's behaviors in the honeypots. It is assumed that the attacker does not identify the existence of the honeypot. Thus, the observed samples at the procedure of 'sense' indicate the attackers' characteristics and contain the correct threat intel-ligence. Then, the defender updates the Q-value at each decision epoch k ∈ {0, 1, · · · , ∞} and then chooses the engagement action a k as shown in the procedures of 'learn' and 'act', respectively. Figure 9: The feedback structure of reinforcement learning methods on SMDP. The attacker's characteristics determine the environmental uncertainties and the samples observed in the honeynet.

Adaptive Alert and Attention Management Strategy against IDoS Attacks
As an analogy to the DoS attacks that generate a large number of superfluous requests to exhaust the limited computing resources in communication systems, IDoS attacks generate a large number of feints to exhaust the operators' limited cognition resources (e.g., attention, reasoning, and decision-making) in human-in-the-loop systems. IDoS attack is a broad class of attacks and poses the following significant security challenges. First, the strategical feint generation exerts additional cognition load to the human operators and exacerbates alert fatigue in the age of infobesity. Second, IDoS directly target human operators in the Security Operating Center (SOC), which is regarded as the cyber immune system. Third, it requires operators of a high expertise level to identify feints and respond to alert timely in complicated systems. Therefore, there is a need to understand this kind of attentional attack, quantify its consequences and risks, and develop new mitigation methods.

Variables
Meaning k, h ∈ Z ≥0 Index for attack stages and inspection stages t k ∈ [0, ∞) Arrival time of the k-th attack Time duration between k-th and (k + 1)-th attack τ h,m Alert's category label at attack stages k ∈ Z ≥0 Expected cumulative cost and expected aggregated cumulative cost v h+1 (x h , a m ),v h+1 (ŝ hm , a m )Learning estimates of ECC and EACC The authors in [24] establish a holistic model in Fig. 10 to formalize the definition of IDoS attacks, evaluate their severity levels, and assess the induced cyber risks. In particular, they model the IDoS attack as stochastic arrivals of feints (denoted by θ F E with probability b F E ) and real attacks (denoted by θ RE with probability b RE ) characterized by the following Markov renewal process. As highlighted by the orange background in Fig. 11, the k-th attack, or equivalently the one at attack stage k ∈ Z ≥0 , happen at time t k , k ∈ Z ≥0 . Let τ k := t k+1 − t k ∈ [0, ∞) be the inter-arrival time between the (k + 1)-th attack and the k-th attack for all k ∈ Z ≥0 . The alerts triggered by these attacks in real-time cannot directly reveal the attack's type denoted by θ k ∈ Θ := {θ F E , θ RE } at all attack stages k ∈ Z ≥0 . However, the alerts can provide human operators with a category label s k ∈ S based on observable features or traces of the associated attacks at attack stage k ∈ Z ≥0 . Manual inspection of these alerts leads to three security decisions: the attack is feint (denoted by w F E ), the attack is real (denoted by w RE ), or the attack's type is unknown (denoted by w U N ). Let w k ∈ W := {w F E , w RE , w U N } denote the human operator's security decision of the k-th alert. Since each human operator has limited attention, frequent alert pop-ups can distract humans from the current alert inspection and result in alert fatigue. To compensate for the human's attention limitation, one strategy is to intentionally make some alerts less noticeable, e.g., without sounds or in a light color. Then, the human can pay sustained attention to the alert currently under inspection. Those inconspicuous alerts receive the security decision w U N and are assigned to other available inspectors with an additional cost of human resources and inspection delays.
The authors focus on the class of periodic Attention Management (AM) strategies, denoted by A := {a m } m∈Z>0 and assume that the human operator can only notice and inspect an alert when it is highlighted. Then, AM strategy a m ∈ A means that the human operator inspects the alerts at attack stages k = hm, h ∈ Z ≥0 . The attack stages during the h-th inspection are referred to as the inspection stage h ∈ Z ≥0 . Then, under AM strategy a m ∈ A, each inspection stage contains m attack stages as shown in the blue background of Fig. 11  Denote c(w k , s k ; θ k ) as the operator's cost at attack stage k ∈ Z ≥0 when the alert's category label is s k ∈ S, the attack's type is θ k ∈ Θ, and the security decision is w k ∈ W. At attack stages where alerts are inconspicuous, i.e., for all k = hm, h ∈ Z ≥0 , the security decision is w U N without manual inspection, which incurs an uncertainty cost c(w U N , s k ; θ k ) > 0. At attack stages of highlighted alerts, i.e., for all k = hm, h ∈ Z ≥0 , the human operator obtains a reward (resp. cost) for correct (resp. incorrect) security decisions. If the human operator remains uncertain about the attack's type after the inspection time τ h,m IN , i.e., w hm = w U N , there is the uncertainty cost c(w U N , s hm ; θ hm ). Define the human operator's consolidated cost at inspection stage h ∈ Z ≥0 as c(x h , a m ;θ h ) := (m − 1)c(w U N , s hm ; θ hm ) + w hm ∈W Pr(w hm |x h , a m ;θ h )c(w hm , s hm ; θ hm ), (19) where the decision probability Pr(w k |s k , a m ; θ k ) represents the probability of human making decision w k ∈ W when the attack's type is θ k ∈ Θ, the category label is s k ∈ S, and the AM strategy is a m ∈ A. With discounted factor γ ∈ (0, 1), defineû(x h , a m ) := E[ and prove the computational equivalency between ECC and EACC under mild conditions. Both ECCû(x 0 , a m ) and EACCū 0 (s 0 , a m ) evaluate the long-term performance of the AM strategy a m ∈ A on average. However, EACC depends on s hm but not on s hm+1 , · · · , s hm+m−1 . Thus, ECC and EACC are referred to as the consolidated and the aggregated IDoS risks, respectively.

Optimal Attention Management Strategy via Reinforcement Learning
Since the parameters of the Markov renewal process are usually unknown, Temporal-Difference (TD) learning are used to evaluate the performance of the AM strategy a m ∈ A based on the inspection results in real-time. Define v h (x h , a m ) as the estimated value ofû(x h , a m ) at the inspection wherex h (resp.x h+1 ) is the observed state value at the current inspection stage h (resp. the next inspection stage h + 1), α h (x h ) ∈ (0, 1) is the learning rate, andĉ h is the observed cost at stage h ∈ Z ≥0 . Alternatively, definev h (x h , a m ) be the estimated value ofū(s hm , a m ) at the inspection whereŝ hm (resp.ŝ (h+1)m ) is the observed state value at the current inspection stage h (resp. the next inspection stage h + 1),ᾱ h (ŝ hm ) ∈ (0, 1) is the learning rate, andĉ h is the observed cost at stage h ∈ Z ≥0 . Finally, the authors provide a numerical case study where category labels s AL , s N L , and s P L in the set S = {s AL , s N L , s P L } represent the application layer, network layer, and physical layer, respectively. Fig. 12a and Fig. 12b illustrate the impact of high and low uncertainty costs c(w U N , s hm ; θ hm ), respectively. If the uncertainty cost is much higher than the expected reward of correct decision-making, then the detailed inspection and correct security decisions are not of priority. As a result, the 'spray and pray' strategy should be adopted; i.e., let the operator inspect as many alerts as possible and use the high quantity to compensate for the low quality of these inspections. Under this scenario,ū(s hm , a m ) increases with m ∈ Z >0 for all s hm ∈ S as shown in Fig. 12a. If the uncertainty cost is of the same order as the inspection reward on average, then increasing m in a certain range (e.g., m ∈ {1, 2, 3, 4, } in Fig. 12b) can increase the probability of correct decision-making and reduce the aggregated IDoS risk. The loss of alert omissions outweighs the gain of detailed inspection when m is beyond that range.  As an analogy to Fig. 5 and 9, we visualize the RL feedback in Fig. 13. Different from the previous two, the cyber system is replaced by the human operator's attention system for alert inspections and responses.

Reinforcement Learning in Adversarial Environment
As more and more networked systems are equipped with RL techniques to improve their performance, security, and resilience, the application of RL also creates opportunities for malicious third parties to exploit. Hence, it is of equal importance to study the security problems of RL itself than RL-enabled networked systems. The successful implement of RL relies on accurate and consistent feedback from the environment, which can be easily guaranteed in a simulated environment. However, in practice, especially in the presence of adversarial interventions, accurate and consistent feedback from the environment is unlikely to be guaranteed. For example, adversaries can manipulate reward signals by performing data injection attacks and prevent an agent from receiving reward signals by launching DoS attacks on the communication channel. Without consistent and/or accurate feedback from the environment, the RL algorithm can either fail to learn a good strategy or be tricked into a 'trap' policy that the attacker aims for. The failure of RL algorithms under adversarial interventions can lead to a catastrophe when RL is applied in critical domains. For example, self-driving platooning vehicles can collide with each other when the measurement data is manipulated [113]; drones equipped with RL techniques can be weaponized by terrorists to create chaotic and vicious situations where they are commanded to crash into a crowd or a building [114].
Hence, it is imperative to study RL under an adversarial environment. In the last five years, there is a surge in terms of the number of papers that studies the security issues of RL [114,115,46,116,117,118,119,120,121,122,123,124,125,126]. The vulnerabilities of RL comes from the information exchange between the agent and the environment. As demonstrated in Fig. 2, the agent receives reward signals and state observations from the environment and sends action protocols to the environment. Suppose the agent is conducting real-time learning by communicating with the environment. In that case, the RL agent is subject to DoS attacks, jamming attacks, spoofing attacks, and data injection attacks that all networked systems might suffer. If the agent learns from batches of stored data from previous experience, the RL agent might suffer from data poisoning attacks, test-item attacks, etc. To understand RL under an adversarial environment, first, we need to understand the adversarial behaviors of the attackers. We need to develope an attack mode that characterizes the attacker's objective, the available attacks, the limitation of attacks, and the information available to the attacker. The second is to understand how the specific attacks on the RL algorithms impact the learning results. To do so, we need to develop metrics that measure the success of the attacks, and the consequences rendered by the attacks. With the understanding of the attack model and its impact on the learning results, the last is to design defense mechanisms to protect RL algorithms from being degenerated. The defensive acts may include the detection and removal of corrupted feedback, resilient RL in the absence of feedback signals, and deploying cryptography techniques to ensure confidentiality, etc. In this section, we discuss the security problems faced by RL and review relevant works based on the three possible signals that the attacker can target: the reward, the sensing, and the actuating signals.

Attacks on the Reward
Reward/cost are the most important signals that the RL agent absorbs in order to learn a proper strategy [43]. The most direct way, perhaps also the easiest way, to trick the RL agent into learning a nefarious strategy is to falsify the reward/cost signals or poison reward/cost data. Huang et al. show that the attacker can alter the learned strategy of many states by poisoning the reward in one state [114]. In the past two years, many research works have been focused on the reward-poisoning attacks on RL [114,115,46,116,121,123,126]. In [115], Wang et al. have investigated the effect of perturbed rewards on a list of RL algorithms. The rewards received by the RL agent are perturbed with a certain probability, and the rewards take values only on a finite set. Here, the rewards are unintentionally perturbed, meaning this work focuses on a robust perspective other than a security perspective.
Huang and Zhu [114] have studied the security perspective of cost manipulation in RL. In this work, the attack model is described by the information available to the attacker, the attacer's actions, and his/her objectives.
• Information the attacker knows: The authors consider an omniscient attacker who knows almost all information regarding the learning process.
• Actions available to the attacker: The attacker can manipulate the cost signals in certain states. Formally, letc ∈ S × A → R + be the cost function manipulated by the attacker. We say the attacker can only alter the cost signals at states S † ⊂ S, then we havec(s, a) = c(s, a) for all s ∈ S\S † , a ∈ A, meaning the attacker cannot manipulate the cost signals at states other than the ones in S † . Here, c is the original true cost function. The authors also consider other types of constraints to describe the limitation of the attacker's power, e.g., the manipulated cost is bounded, i.e., c −c ≤ B, where · is an appropriate norm. The bound of the manipulation π † A nefarious strategy learned from manipulated signals π * A optimal policy learned from accurate signals β Discount factor Q * Optimal Q function learned from manipulated signals r 0 t True reward signal at time t D := (s t , a t , r 0 t , s t ) Training set r := (r 0 , · · · , r T )

R
Constructed reward function using the training set P Constructed transition probabilities using the training set • The objective of the attacker: The objective of the attacker is to trick the RL agent into learning a nefarious strategy π † instead of the original optimal strategy π * .
Among many RL algorithms, Huang and Zhu have studied Q-learning algorithm and its limiting behavior under the manipulated cost signals. The authors develop some fundamental limits regarding the attack model. We highlight a few results in this section.
Theorem 1 ( [114]). Let Q * denote the Q-factor learned from Q learning algorithm (1) with the true cost signals c andQ * be the Q-factor learned from the manipulated cost signalsc. Then, we have where β is the discount factor of the underlying MDP problem.
Theorem 1 states that manipulation on cost c using a small perturbation does not cause significant changes in the limit point of the Q learning algorithm. This result indicates that the attacker cannot cause a significant change in the learned Q-factor by just a tiny perturbation in the cost signals. This is a feature known as stability observed in problems that possess contraction mapping properties. One conclusion we can make from (23) is that if the attacker wants to successfully make the RL agent learn the nefarious strategy π † , the attacker has to manipulate the cost signals with magnitude more significant than a certain value: where V π = {Q ∈ R |S|×|A| : Q(s, π(s)) < Q(s, a), ∀s ∈ S, ∀a = π(s)}.
Huang and Zhu have also shown that to make the agent learn the nefarious strategy π † , the manipulated cost signal has to satisfies where 1 s ∈ R |S| is a |S|-dimensional vector whose elements are all zero except that the s-th element is 1, P sa = (p(s, 1, a), p(s, 2, a), · · · , p(s, |S|, a)) T ∈ R |S| , and P π ∈ R |S|×|A| is a matrix whose ijcomponent is [P π ] i,j = p(i, j, π(i)). Here, p(s, s , a) is the probability that given current state s and current action a, the next state is s , which characterizes the transition kernel of the underlying MDP problem. If the attacker manipulates the cost signals such that (24) is satisfied, the agent will be misled into the nefarious strategy π † . The conditions in (24) can then be utilized by the attacker to design an optimal manipulation of the cost signals that optimizes the cost of manipulating the cost signals incurred to the attack. Huang and Zhu have also shown theoretically and numerically that the attacker can achieve his/her objective by manipulating the cost signals in a small subset of states.
In [123], Ma et al. have studied a similar problem for model-based RL. The attacker poisons the reward signals stored in a dataset that is used to train an RL agent. The attack model is given as • Information the attacker knows: The attacker has access to the training set D = (s t , a t , r 0 t , s t ) t=0:T −1 , in which r 0 t is the true reward signal the agent has at time t. The attacker also knows the model-based RL learner's algorithms.
where T s,a = {t|s t = s, a t = a} is the time indices of all training data points for which action a is taken at state s. By solving (25), the attack can mislead the RL agent into learning the strategy π † while minimizing his/her own cost of manipulating the reward data. Ma et al. have showed the existence of the optimal solution to (25) and provided a bound on the optimal cost that the attacker needs to pay to achieve the strategy π † . Ma et al. have applied the attacking strategy given by the solution of (25) to an RL-based linear-quadratic regulator (LQG) problem. In Fig. 14 (b), the upper figure shows the true reward data and the poisoned reward data, and the bottom figure shows the difference between the two rewards. At each time step, the true reward is only poisoned slightly. However, the sight poisoning can drive the vehicle into a designated position, as we can see from Fig. 14 (a).
As the dataset becomes large D, i.e., T increases, solving (25) becomes prohibitive. Zhang et al. have proposed an adaptive reward-poisoning attacking scheme that focuses on online RL attacks. The reward poisoning is done on the fly. Hence, their attacking strategy does not require solving a large-scale optimization problem [116]. The authors demonstrated how easily an attacker could trick the agent into a designated target in a grid world problem.
As the topic is an emerging area, most works have focused on discussing the vulnerabilities of RL instead of designing defensive mechanisms. Very few works have been studied the defense against reward poisoning attacks [121]. In [121], Banihashem et al. have designed a defense approach to make sure the RL agent is robust against reward poisoning attacks.

Attacks on the Sensing
Attacks that target the sensing components, including sensors and the communication channels in CPS control systems, have been well-studied [63,127,128,129,54]. RL-based systems also share the same vulnerabilities with CPS systems -the observation may be delayed, inaccurate, and/or missing. One difference between sensing attacks on RL and CPS is that sensing attacks can happen during the training phase, testing phase, and/or execution phase. Sensing attacks during the training phase can render the RL agent learn a 'bad' policy that leads to dangerous behavior in the testing phase. Attacks in the testing phase can provide false performance evaluation to the RL agent. Attacks during the execution phase are more similar to attacks in CPS control systems, in which controllers/agents receive real-time measurements and send real-time commands. Hence, during the execution phase, any delay, inaccuracy, or absence of the observation can cause malfunctioning of the system.
Offline RL can tolerate a certain degree of delay of the feedback signals, including reward signals, state signals, since offline RL starts after gathering a collection of data points. However, online RL can be sensitive to delays. In 2003, [130] considered delayed state observation, in which the state observation is delayed, and the RL agent needs to select an action without any knowledge of its current state. For a fixed delay d, the authors formed a non-delayed MDP whose size grows exponentially in d. In [131], Lancewichi et al. have studied the delayed feedback in online reinforcement learning algorithms. In their work, an attacker can strategically choose the delay d t , meaning that the cost signal and the state observations are only available at the end of episode t + d t . The authors proposed a policy optimization-based algorithm that can achieve near-optimal high-probability regret of orderÕ( where T is the number of episodes and D = t d t the total delay. More recently, Ramstedt et al. [132] proposed the delay-correcting actor-critic algorithm that can achieve better performance in environments with delays. RL can be an easy target of DoS/jamming attacks and spoofing attacks and can suffer a significant loss if the attacks are successfully launched. Take the autonomous vehicle as an example. If the state is the location of the RL agent measured by GPS coordinates, spoofing of GPS signals by the attacker can lead to incorrect navigation policy and hence, a collision. Although RL suffers a high risk and the potential loss of being attacked, very few works have focused on RL with missing or inaccurate observations from a security perspective. Hence, the attack models and the defense mechanisms about DoS attacks and spoofing attacks on RL sensing remain open for future studies. RL agents observe the environment via sensors. Under DoS attacks, the state observations cannot reach the RL agent, and the agent becomes unmindful of the states. For remote sensing, jamming attacks can be launched on the communication channels. For in-house sensing, an example is sequential blinding of the cameras in an RL-based autonomous vehicle via lasers, leading to a collision. On behalf of the attack, researchers can address the following interesting questions: when and where to launch DoS/jamming attacks to create the most damage to the RL agent, whether the attacker can trick the RL agent into a nefarious policy only through jamming attacks, etc.; on behalf of the agent, researchers can design a resilient RL algorithm that is less sensitive to missing state observations [127], employ prediction methods when state observations are missing [133], or study the effect of DoS attacks on existing RL algorithms. Attacks can also manipulate the sensing signals by spoofing attacks. The problem for the attacker can be to design a manipulation strategy. The manipulation strategy can be characterized by a map K that maps a state s into a probability simplex over all possible states ∆S = {p o (s), s ∈ S}. For example, ∆S = K(s t ) indicates that given the true state s t at time t, the RL agent will instead observe the state s t ∈ S with probability p o (s t ). The map K is similar to the concept of observation kernel in partially observable Markov decision process [134] except that now the map K is crafted by the attacker to mislead the RL agent and achieve his/her malicious purposes.
One concept that is close yet different to sensing attacks on RL is environment poisoning attacks, which is the focus of many works regarding the security of RL [124,119,117]. In environment poisoning attacks, the attacker can perturb the environment, i.e., the transition probability in the MDP. As is demonstrated by Behzadan and Munir in [113], through sequential reconfiguration of obstacles on the road, an attacker can manipulate the trajectory of an RL-based autonomous vehicle in the testing phase. In 2020, Rakhsha et al. [117] study the problem of how to teach a policy to the RL agent through transition dynamics poisoning. We describe the attack model proposed by the authors using the following description: • Information the attacker knows: The attacker is omniscient who knows almost all information including the original MDP.
• Actions available to the attacker: The attacker can turn the original transition dynamics denoted by P into the manipulated transition dynamicsP .
Here, π{s, a} is a neighbor policy of the policy π defined as π{s, a}(s ) = π(s ) s = s, a s = s.
The neighbor policy π{s, a} is almost the same with policy π except that at state s, the neighbor policy points to action a. Here, µ π is the stationary distribution given the transition probability P and policy π, which satisfies µ π (s) = s p(s , s, π(s )) · µ π (s ).
In the optimization problem (26), the first three constraints are the conditions needed to be satisfied to make sure the agent learns the policy π † . The last constraint specifies how much the attacker can decrease the original values of transition probabilities, where δ ∈ (0, 1]. The authors later demonstrated that using the attacking rule obtained from (26), the attacker can achieve his objective by only altering a small number of transition probabilities [117]. However, this approach faces scalability issues and the assumption that the attacker knows the MDP is a stringent one, which makes the attack a white-box attack. In 2021, Xu et al. [124] studies environment-dynamics poisoning attacks at training time which expands the attacks to black-box attacks. The authors propose an attack model that prompts the momentary policy of the agent to change in the desired manner with minimal dynamics manipulation. More recently, Sun et al. [119] proposed an attack model that does not require any knowledge of the MDP. The attack model follows the so-called vulnerability-aware adversarial critic poisoning strategy. The authors demonstrated that the attacking strategy successfully prevents agents from learning a good policy or mislead the agents to a target policy with low attacking effort.

Attacks on the Actuator
Many works have been done to study attacks on the actuators or the communication channel between the actuators and the controllers for CPS control systems [135,136,137,138,139,140]. However, there are very few studies investigating attacks targeting the RL agent's actuators. RL agents influence their environments by performing actions a t via actuators. To ensure that the action is timely and accurately applied to the environment, the RL agent must guarantee first that the action command is timely and accurately transmitted to the actuator in the field. Second, the actuator executes the received commands in a timely and accurate manner. However, the attacker may exploit these vulnerabilities. Suppose the attacker can launch jamming attacks on the communication channel or manipulate the actuator. In that case, the actual action performed will be different from the one chosen by the RL agent, and hence the observed experience is corrupted. The learning results will also be corrupted if the RL agent learns from the corrupted experience without noticing the attacks. Future works need to focus on investigating how attacks on the actuator affect the performance of the RL agent in training, testing, and implementing phases.

Discussion and Future Outlooks
While there is vast literature on IoT and CPS security issues, little is known about the security issues of RL. However, security issues of RL exist and are getting increasingly significant as the application domain of RL widens. This section reviews the literature that studies RL under an adversarial environment, where three vulnerabilities are targeted by the attacker: the reward signals, the sensing, and the actuating. These are the three signals that need to be exchanged timely and accurately between the agent and its environment, which becomes the vulnerabilities of RL-enable systems that the attacker can exploit. Unlike real-time systems such as CPS control systems, attacks on different phases of RL may serve different purposes. Attacks during the training phase can render the RL agent fail to learn a 'good' policy. Attacks in the testing phase can provide false performance evaluation to the RL agent. Attacks during the execution phase are more similar to attacks in CPS control systems, in which controllers/agents receive real-time measurements and send real-time commands. Hence, during execution phase, any delay, inaccuracy, or absence of the observation can cause malfunctioning of the system.
Within relatively small literature on the security of RL, most works have focused on the attacks on reward signals, and few papers have investigated the poisoning attacks on the transition dynamics. They mainly focus on deceptive attacks that falsify the values of either the reward signals or the transition probabilities. To the best of our knowledge, very few works are dedicated to DoS/jamming attacks on either the reward, the sensing, or the actuating. Hence, we believe there is an emerging research opportunity since DoS/jamming attacks are considered as the most common attacks that could happen to a networked system [54]. Another gap that needs to be narrowed is the attacks on sensing and actuating for RL-enabled systems. We notice that few works have only addressed the problem of missing or noised sensing/actuating signals from a non-security perspective that achieves robustness for the RL-enabled system. Only with a good understanding of how different attacks affect the RL, one can further safeguard our RL-enabled systems.

Conclusions and Discussions
Cyber-resilient mechanisms (CRM) complement the imperfect cyber protections by strategically responding to unknown threats and maintaining the critical functions and performances in the event of successful attacks. A CRM can be viewed as a feedback system that consists of three critical components: sensing, reasoning, and actuation. Sensing aims to acquire information about the system as well as the footprint of the attacker. Reasoning builds on the acquired information to infer the attack behaviors and the design of the optimal resilience strategies. Actuation reconfigures the system according to the optimal strategy by adapting the system parameters and attributes to unknown threats. The sensing-reasoning-actuating feedback loop establishes an adaptive and dynamic system architecture for cyber resilience.
Reinforcement learning (RL) is a suitable data-driven framework for cyber resilience which implements the feedback-system architecture. The update of the system reconfiguration strategies dispenses with the accurate system models and relies on only the observations of the system state and its associated cost at each iteration. We have classified the design of RL-based CRM (RL-CRM) based on the type of vulnerabilities the system aims to mitigate. We have demonstrated that the RL-CRM has enabled an adaptive and autonomous way to configure moving target defense, engage attackers for reconnaissance, and guide human attention to mitigate visual vulnerabilities. We have learned from the literature that posture-related defense technologies are mature, while the mitigation solutions for information-related and human-induced vulnerabilities are still underdeveloped. New advances are needed to detect and deter deceptive attacks. Besides compensating for information disadvantage, the defender can proactively create uncertainties and increase the attack cost by designing defensive deception. To defend against human-induced vulnerabilities, we have observed the need for new designs in human-machine interactions. Besides indirectly designing corrective compensation for human-induced vulnerabilities, we also need to focus on (non)monetary incentives supported by human-study evidence and human-assistive technologies to elicit desirable human behaviors.
We have discussed the RL solutions to mitigate vulnerabilities of the cyber systems and the vulnerabilities of the RL itself. Vulnerabilities of RL exist and are growing increasingly significant as the application domain of RL broadens. The RL agent exchanges three kinds of time-sensitive signals with the environment, including the rewards, the state observations, and the action commands. The three signals become the vulnerabilities of RL-enable systems that the attacker can influence. We have reviewed the current state-of-the-art involving the attacks on the three kinds of signals. These works show that under proper conditions, a small malicious perturbation of the feedback signals can mislead the RL agent to learn any policy desired by the attacker. These results serve as a warning about the seriousness of the security issues of RL-enabled systems.
The study of vulnerabilities of RL from a security point of view is an emerging topic. We have seen a surge of publication on the topic in the last two years. Most works have focused on the attacks on reward signals, and a few papers have investigated the poisoning attacks on the transition dynamics. However, almost no works have studied the attacks on sensing and actuating for RL-enable systems, which is a considerable research gap that needs to be filled. Furthermore, current literature primarily focuses on deceptive attacks that falsify the values of either the reward signals or the transition probabilities. Since DoS/jamming attacks are considered the most common attacks that could happen to a networked system, more works should be dedicated to DoS/jamming attacks on either the rewards, the state observations, or the actions commands.

Feedback Architectures
The RL-CRMs presented in this work follow the standard feedback architecture. This architecture can be enriched and extended to several more sophisticated ones. One is the nested feedback loops, where one RL feedback loop is coupled in another RL feedback loop. This architecture is useful to separate and then fuse the learning of distinct system components of the cyber system. For example, one RL feedback loop is used to acquire the attack footprint and learn its intent and capabilities. In contrast, the other feedback loop is used to acquire information regarding its system state. The two feedback loops can be fused for making online defense decisions in response to an unknown threat.
Another architecture is a mixture of feedback and open-loop structures. Leveraging the ideas from moving-horizon control and estimation, the CRM can make a moving-horizon plan by looking N stages into the future and optimizing the cyber resilience for the current stage and N stagesto-go. This approach would require an open-loop prediction of the system under the attack and feedback-driven sensing of the environment and reasoning of the optimal moving-horizon resilience strategies. This architecture enables a look-ahead strategy that can prepare for a sequence of forthcoming events and improve resilience.
A centralized CRM would not be sufficient for large-scale cyber systems to provide a timely response since the amount of data would increase exponentially with the size, and so does its processing time. A distributed CRM can locally monitor a subsystem and coordinate with other CRMs to achieve a global CRM performance. In doing so, we can achieve local cyber resilience for the subsystem as well as the global resilience of the entire system. When one subsystem is attacked and fails, the remaining subsystems can adapt to it, maintain their local functions, and recover from the failure of one subsystem. This decentralized mechanism aligns with the recent advances in multi-agent control systems [141], distributed control theory [142], and game theory [143,144]. They can provide a theoretical underpinning for the design of distributed cyber-resilient systems.

Reinforcement Learning
From the three applications of RL-CRM designs, we have identified several research challenges to be addressed in the future works in this direction. First, it is important to deal with system and performance constraints in the learning process. Cyber systems have many system constraints that need to be taken into account explicitly. For example, certain addresses or functions that are not allowed or undesirable when configuring the moving target defense. The performance of the cyber systems can impact the performance of the physical systems that they serve. Hence, the requirement on the physical system performance naturally imposes a constraint on the performance of the cyber systems. Hence, the CRM would need to adapt and respond while satisfying these performance constraints.
A second challenge is to improve the learning speed. The goal of CRM is to restore the cyber system after an attack. Fast learning would enable a speedier and more resilient response to the attack. To achieve it, we would need to resort to control-theoretic ideas, such as optimal control [145] and adaptive control theory [146], and leverage recent advances in reinforcement learning to speed up the convergence rate or improve the finite-time learning performances. A third challenge is to deal with the nonstationarity of the cyber systems. The classical RL algorithms assume that the environment is stationary and ergodic. In many cybersecurity applications, the systems are nonstationary. The system parameters and attributes change over time. For example, the attack surface may grow when the system is connected with other nodes or used by new users. There is a need to develop nonstationary RL schemes for cyber systems that guarantee performance in a finite horizon.

Emerging Applications
Beyond the applications presented in this work, there are many emerging ones where RL-CRM can play a pivotal role in improving their security stature. One crucial area is the cyber resilience for wireless communication systems under jamming and spoofing attacks. Wireless devices can prepare for a jammer with limited power by introducing redundant links and channels [147,148]. An RL-CRM can learn from the communication and the environment to reconfigure adaptively their routes, channels, or access points when they are subject to jamming attacks [149,150]. The devices can also leverage moving target defense to create an RL-CRM [151] to attract jammers to attack a fake route in order to protect the legitimate communication.
The concept of cyber resilience is also applicable to safeguard Cyber-Physical Systems (CPS) and the Internet-of-Things (IoT) networks. The impact of the successful attacks on CPS and IoT systems is not limited to cyber performance. They can create catastrophic consequences such as the meltdown of nuclear energy systems [152,153], the blackout of the electric power grid [154,155], the disruption in gas pipelines [156], and casualties in road accidents [157]. The CPS resilience deal with resilience at both cyber and physical layers of the system. The physical-layer resilience takes into the physical-layer performances and requires a learning-based control system to respond and restore the degrading performance resulting from successful attacks. Cyber resilience and physical resilience are interdependent. An effective CRM would restore the cyber performance and thus reduce the impact of the threat on the physical layer of the system, therefore facilitating physical resilience. There is a need to create a cross-layer resilience design framework [158] to provide a holistic view toward CPS resilience and reduce cross-layer cascading failures [159].
Cyber insurance is an alternative paradigm to improve cyber resilience from a non-technical and economic perspective. Cyber insurance aims to transfer the unpreventable and unmitigable risks to a third party to reduce the economic impact of cyber attacks on an organization. As the attacks are becoming financially motivated, e.g., the rise of ransomware, cyber insurance is an important mechanism that needs to be integrated into the RL-CRM to mitigate further the economic loss induced by the loss of cyber performance. Interested readers can refer to [160,161,162] for the discussions on cyber insurance as a tool for cyber risk management.