{'fraction_non_alphanumeric': 0.0362400906002265, 'fraction_numerical': 0.007159035754732244, 'mean_word_length': 4.623152149192632, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Adversarial Reinforcement Learning under Partial Observability in Software-Defined Networking

Recent studies have demonstrated that reinforcement learning (RL) agents are susceptible to adversarial manipulation, similar to vulnerabilities previously demonstrated in the supervised setting. Accordingly focus has remained with computer vision, and full observability. This paper focuses on reinforcement learning in the context of autonomous defence in Software-Defined Networking (SDN). We demonstrate that causative attacks---attacks that target the training process---can poison RL agents even if the attacker only has partial observability of the environment. In addition, we propose an inversion defence method that aims to apply the opposite perturbation to that which an attacker might use to generate their adversarial samples. Our experimental results illustrate that the countermeasure can effectively reduce the impact of the causative attack, while not significantly affecting the training process in non-attack scenarios.


Introduction
The Adversarial Machine Learning [1,10,3,28] literature has demonstrated that machine learning models are vulnerable to both exploratory (test-time) and causative (training-time) attacks. These attacks are typically crafted by applying calculated perturbations to the test or training instances, in order to either cause misclassification or poison the training process. More recent studies [11,2,7,25] have shown that similar attacks can also be effective against reinforcement learning (RL) algorithms. * This research was supported under the Defence Science and Technology Group's Next Generation Technologies Program.
Unlike previous work that mainly focuses on the vision domain, Han et al. [7] analyse how reinforcement learning agents react to different forms of poisoning attacks in the context of autonomous defence in Software-Defined Networking (SDN) [20]. In their experiments they train an RL agent to inform the decisions of an SDN controller seeking to prevent an attacker from propagating through a network. They investigate the effect of an attacker poisoning the RL training process. Section 2 provides a more detailed description.
The work of Han et al. [7] has a number of limitations: (1) full observability of the (network) states is assumed in the analysis, which is often not the case in real-world situations, especially for the attacker; (2) while an important topic, treatment of RL defence mechanisms is preliminary; and (3) the experiments are performed on a relatively small network.
In this work, we address these limitations by: first imposing partial observability for the attacker. Since it is unlikely that the attacker can map out the entire network topology, we consider the scenario where the defender has full observability of the network, but the attacker only knows part of the topology. Fig. 1 depicts the example network studied in this paper. Indeed a key motivation for moving beyond computer vision is novel properties of the adversarial setting.
Second, we consider a much larger network with 100 nodes and 172 links. As shown in Fig. 1, the attacker has an initial foothold of a handful of compromised nodes, and aims to propagate through the network to take control of a specific node corresponding to the critical server, which in response can be migrated by the defender to some pre-determined alternate nodes. Under this setup, the defender trains a reinforcement learning agent to (1) protect the critical server from being compromised, and (2) maintain the network functionality as much as possible, i.e., maximise the number of nodes that can reach the critical server. On the other hand, the attacker only  has partial observability, which restricts their action set: they cannot compromise an adjacent node unless the link to the node is known.
Third, we propose a new inversion defence method to counteract the causative attack on reinforcement learning algorithms. Our experimental results suggest that the approach introduced in [7] does not work well in our setup (Fig. 1). Instead, we design a method that attempts to undo how attackers poison the training process of the RL agents. We demonstrate the effectiveness of the new defensive algorithm, and show that it has limited impact in non-attack scenarios.
The remainder of this paper is organised as follows: Section 2 summarises the problem of applying reinforcement learning for autonomous defence in computer networks; Section 3 introduces the causative attack via state perturbation and Section 4 the defence mechanism; Section 5 presents the experimental verification; Section 6 reviews previous work in adversarial machine learning; and finally Section 7 concludes the paper and offers directions for future work.

Problem: Reinforcement Learning For Autonomous Network Defence
We now overview the problem of autonomous defence in computer networks using reinforcement learning.

Background on Reinforcement Learning
Reinforcement learning [27] deals with a sequential decision making problem where an agent interacts with the environment to maximise its rewards. At each time step t, the agent (1) receives an observation s t of the environment; (2) takes an action a t based on its policy π, which is a mapping from states to actions; and (3) obtains a reward r t based on state s t , action a t , and the environment's transition to a new state s t+1 . The goal of the agent is to maximise its cumulative rewards, i.e., R t = ∞ τ=t γ τ−t r τ , where γ ∈ (0, 1] is a discount factor which affects the present importance of long-term rewards. We focus our experiments on two widely used RL algorithms-Double Deep Q-Networks (DDQN) [8] and Asynchronous Advantage Actor-Critic (A3C) [17]-and transfer of attacks between them.

Autonomous Network Defence with Reinforcement Learning
In a computer network of |N | nodes, N = {n 1 , n 2 , ..., n | N | }, and |L| links, L = {l 1 , l 2 , ..., l |L | } (e.g., Fig. 1): N D ⊂ N is the set of critical servers to be protected (one or more blue nodes), N M ⊂ N is the set of possible migration destinations for n ∈ N D (one or more green nodes), and N A ⊂ N is the set of nodes that have been compromised (red nodes). In addition, while the defender knows all the nodes and links, the attacker is only able to map out a subset of them, i.e., The attack scenario we consider is a cyber attack against the network infrastructure. Here, the attack spreads through the network, and aims to take control of the critical servers (note that here we assume that the attacker has to compromise all nodes on the path). However, they can compromise a node n only if there is a link l ∈ L O between n and a compromised node n ∈ N A . That is N A keeps expanding as the attack proceeds.
In order to protect the critical servers from being compromised, the defender trains an RL agent that: 1. Monitors the system state. The system state is represented using a binary feature representation. The state representation has a number of bits equal to the sum of the number of nodes and number of links in the network. A bit corresponding to a node is 0/1 to represent whether that node is un/compromised. A bit corresponding to a link is 0/1 to represent whether that link is down/up.

2.
Makes a decision on the appropriate action to take when in a given system state. The actions that are available comprise: (i) isolating and patching one node; (ii) reconnecting one node and its links; (iii) migrating the critical server and selecting the destination; and (iv) taking no action. Note that for actions (i) or (ii), only one node can be isolated or reconnected each action cycle time; The reward function that the RL agent is trained on is based on (i) whether any critical server has been compromised; (ii) the number of nodes reachable from the critical servers; (iii) the number of compromised nodes. In addition, another two factors are also taken into consideration: (i) the migration cost and (ii) the validity of an action, e.g., if a node has already been isolated, it cannot be isolated again.
Under the described RL setup, we train a DDQN [8] agent with Prioritised Experience Relay [26]) and an A3C [17] agent. Without tampering, both agents are able to learn the optimal policy: the optimal policy for our example network being to isolate nodes in the order However, the above cyber attack scenario and resulting trained RL agents leave important questions unanswered: If the attacker has the ability to poison the training process, can both agents still identify the optimal actions? What can the defender do to mitigate attack impact? We seek to address these questions.

Partially-Observable Attacks on RL by State Manipulation
In order for RL techniques to be successfully applied in autonomous cyber defence, it is crucial to analyse susceptibility of RL agents to potential causative attacks. We have investigated the following attack mechanisms: (1) tampering with a small number (e.g., 5%) of rewards to maximise the defender's loss. Specifically, gradient information is used to select which rewards to tamper with; (2) random perturbation of the observed states; (3) manipulating the states to minimise the defender's rewards; and (4) manipulating the states to minimise the probability of taking the optimal action. In our preliminary unreported experiments we found that (4) was the most effective and hence we subsequently use it as the attacker's strategy. We focus on the scenario where the attacker tampers with the states observed by the RL agents, so that the trained model learns sub-optimal actions. Specifically, suppose that the agent observes an experience (s, a, s , r) without any attacks, where s is the current system state, a is the action taken by the agent, s is the new state, and r is the reward. When the system reaches the new state s , the agent would continue to take the next optimal action a . The attacker can counteract this by introducing false positive (FP) and false negative (FN) readings in s , meaning that uncompromised (compromised) nodes will be reported as compromised (uncompromised) to the defender. Consequently, the agent observes (s, a, s +δ, r ) (where δ represents the FP and FN readings) instead of (s, a, s , r), and hence may not take action a next.
The key issue here is how the attacker chooses the nodes to manipulate. We consider the following strategy: 1. Against the DDQN agent: loop through all observable nodes to find δ that minimises the Qvalue of the optimal action a for state s + δ, i.e., argmin δ Q(s + δ, a ); 2. Against the A3C agent: loop through all observable nodes to find δ that minimises the probability of taking the optimal action a for state s + δ, i.e., argmin δ π(a |s + δ).
We next abstract the threat model for adversarial learning in SDN as follows: Black-box approach. The attacker does not have access to the defender's training model as per our partial observability assumption. In other words, this constitutes a form of black-box attack, which means the attacker needs to train their own surrogate model first, based on the partial topology visible to them.
Limited choice of potential false positive and false negative nodes. It is unlikely that the attacker can falsify the state of all observable nodes. Therefore, we limit the nodes whose states can be perturbed by the attacker. Section 5 further explains how these nodes are selected.

Limits on the number of false readings per time step.
In our experiments, the number of false positive and false negative nodes that can be introduced per time step are no more than two in each case.
Our view is that this model of attacker information/control is a key point of interest in exploring domains beyond computer vision. Algorithm 1 details this attack against DDQN. The algorithm for attacks against A3C is similar and so is omitted.

The Inversion Defence Mechanism
For the defender we propose a countermeasure that generates training instances by applying a perturbation counter to simulated adversarial samples.
We aim to design a defence mechanism that (1) effectively mitigates the impact of the above causative attack, (2) requires minimum knowledge of the attacker, and (3) does not affect the training when there is no attack.
Since the attacker adds false readings δ into the observed states, can δ be reversed? If the defender knows the nodes that are visible to the attacker, limits on the FP & FN nodes, and the number of such nodes, then they may find these false readings, by solving the inverse problem of how the attacker generates the adversarial samples: while the attacker receives (s, a, s , r), and loops through all observable nodes to find δ that either minimises the Q-value or the probability of action a for state s + δ, the defender receives (s, a, s + δ, r ), and through the same nodes may find δ maximizing argmax δ Q(s + δ + δ , a ) for DDQN, and argmax δ π(a |s + δ + δ ) for A3C. In other words, δ = −δ. However, the attacker's partial knowledge of the network topology, the limits on the choice of FP & FN nodes, and the number of false readings per time interval/step are not made known to the defender. Therefore, we propose that instead of looping through the nodes observable to the attacker, the defender necessarily goes through all network nodes to find δ . In addition, we also test using a different number of false readings in our experiments (Section 5). δ obtained in such a way may not exactly match δ, and the defender can choose to either keep both (s, a, s + δ, r ) and (s, a, s + δ + δ , r ), or just (s, a, s + δ + δ , r ).
This method does not make any assumptions about the attacker, except that they falsify the states of certain nodes. However, as demonstrated by the experimental results in Section 5, the method is effective against the causative attack, and it does not prevent the agent from learning the optimal actions in the non-attack scenario.

Experimental Results
We next introduce our experimental setup, present how DDQN and A3C agents are affected by causative attacks, and demonstrate effectiveness of the proposed defence.

SDN Experimental Environment
In order to better cope with today's dynamic and high-bandwidth traffic, Software-Defined Networking (SDN) [20] is designed as a next-generation tool chain for network management. Not only does it provide a centralised view of the whole network, the core componentthe SDN controller-receives network requirements from online applications via northbound APIs, translates them into low-level controls, and sends them to the infrastructure layer (switches, etc.) via southbound APIs. In this way, network control is decoupled from forwarding functions, making the controller directly programmable. It is thus convenient to monitor and reconfigure network resources. In this paper, we choose OpenDaylight [16], the most popular open-source SDN controller available.
Specifically, we use Mininet [21], a popular network emulator, to create the network with 100 nodes and 172 links as shown in Fig. 1. Once the network is created, OpenDaylight is added as the controller. It provides APIs for the RL agent to retrieve network information and take different kinds of operations as defined in Section 2.

Causative Attacks via State Perturbation
As described earlier in Section 3, the attacker needs to: (1) train their own model-we achieve this by training a DDQN agent using the partial topology visible to the attacker. The model is used as the surrogate to attack both of the defender's models (i.e. both DDQN and A3C agents); (2) limit the nodes that they can perturb (this is an appropriate threat model-even if the attacker can map out part of the network topology, it is very unlikely that they can manipulate the states of all those nodes). We run the attack by adding one FP and one FN per time interval/step but without any limits on the choices of FPs and FNs. In this way, we are able to find the nodes that are most frequently selected as FPs and FNs, i.e., lists L F P and L F N in Algorithm 1. Note that the nodes in L F P and L F N are different for the DDQN and A3C agents. The attacker is only allowed to manipulate the states of these nodes; The results demonstrate that (1) the causative attack designed in Algorithm 1 is effective against both agents when there is no form of defence-a significant percentage of attacks either cause the critical server to be compromised (the red bars), or cause fewer nodes to be preserved (the blue bars). Note that this also demonstrates the existence of transferability between RL algorithms [22]-attackers do not need to have knowledge of the defender's model and hence attempting to keep the model secret is not an effective countermeasure against adversarial machine learning attacks; (2) given the same number of false readings per time step, the stricter the limits on the choices of FPs and FNs, i.e., the smaller |L F P | and |L F N | are, the less powerful the attacks are-not only do the limits restrict which nodes can be manipulated, they decrease the number of steps that are poisoned in each training episode; (3) interestingly, if we compare the second and fourth bars in both Figs. 3a & 3b, when |L F P | = 6, adding one FP & one FN per time step is more effective than adding two FPs & two FNs per time step. This is because more training steps are likely to be poisoned in the former case given that |L F P | is the same.
In the next section, we test our proposed countermeasure against the most powerful form of attack as illustrated in Fig. 3, where |L F P | = 8, |L F N | = 2, and two FPs & two FNs are added per time step.

Countermeasure
Our inversion defence method only assumes that attackers perturb the states of a certain number of nodes in each training step, and aims to identify & revert the manipulations. However, the defender has to loop through all the nodes rather than the nodes in L F P & L F N , and has to estimate the number of false readings added per step.
Specifically, four scenarios are investigated: In the first three scenarios, the attacker adds two FPs & two FNs per training step, and |L F P | = 8, |L F N | = 2 (i.e., the most powerful form of attack studied in the experiments), while the defender assumes that there are (1) one FP & one FN, (2) two FPs & two FNs, (3) three FPs & three FNs per training step. In the last case, the defender assumes that two FPs & two FNs are added per time step, but in fact there is no attack.
The first three scenarios investigate the situations where the defender either does or does not know the limit on the number of false readings added per time, while the last scenario is designed to study whether the normal learning process will be impacted when the defender falsely assumes the presence of an attack.
Comparing the rightmost bars in Figs. 3a & 3b and the left three bars in Figs. 4a & 4b, we can see that the proposed defence method can effectively mitigate the impact of the causative attacks-the percentage of experiments where the critical server is compromised drops from al-  most 100% to less than 30% on average. In addition, the two rightmost bars in Fig. 4 also indicate that in most cases the defence method will not prevent the agent from learning the optimal actions when there is no attack-in all the cases represented by the blue bar in Fig. 4a, only one less node is preserved.
A disadvantage of the inversion defence method is that it significantly slows down the training process, as it is time-consuming to loop through all the nodes to find the potential FPs and FNs. We aim to improve the performance in our future work.

Related Work
This section first summarises adversarial machine learning against supervised classifiers, and then reviews recent work on similar attacks against reinforcement learning models. Finally, we discuss existing defence mechanisms.

Adversarial Machine Learning
Adversarial machine learning aims to minimise the modifications to the input, i.e., either the test instance or the training sample, to cause a malfunction of the machine learning model. Biggio et al. [3,4]  of evading a machine learning classifier as optimisation of the model's continuous scores, and use gradient descent to generate adversarial samples. Szegedy et al. [28] highlight the observation that modifications imperceptible to humans can cause deep neural networks (DNNs) to misclassify, and they design the Fast Gradient Sign Method [6] for the attack. Since then a number of different methods for creating adversarial samples have been proposed [19,23,24,18,5,14], among which the C&W attack [5] is empirically the most efficient exploratory attack so far.

Adversarial Reinforcement Learning
More recent work has shown that reinforcement learning models are also vulnerable to the above attacks against classifiers. For example, Huang et al. [11] demonstrate that both white-box and black-box attacks using the Fast Gradient Sign Method [6] are effective against deep RL.
Behzadan & Munir [2] were the first to investigate causative attacks against RL agents. They show how adversaries can perturb the observed state, in order to prevent the DQN agent from learning the correct policy. Specifically, the perturbation is generated using both the Fast Gradient Sign Method and the Jacobian-based Saliency Map Attack [23].
Lin et al. [12] propose two types of attacks against deep RL: (1) strategically-timed attack, which aims to decrease the number of time steps to launch the attack (i.e., manipulate the state). It estimates when an adversarial sample will be effective, and uses the C&W attack [5] to perturb the corresponding states; (2) enchanting attack, which aims at misleading the agent to a specific state. It uses a sampling-based cross-entropy method to find a sequence of actions that will guide the agent to the target state, and progressively craft the states so that the agent will always take the next required action. Pattanaik et al. [25] show that even the naïve attack, that is, adding random noise into the current state, is effective against deep RL-this is contrary to our experimental findings. However, our scenario is different to that described by the authors, including the dimensions of the state, the action space, and they design a gradient based attack that aims to maximise the probability of taking the worst possible action.

Existing Defence Mechanisms
Generally speaking, existing defence methods against adversarial machine learning can be categorised into two classes: (1) data-driven defence, which either filters adversarial samples, injects adversarial samples into training-a.k.a., adversarial training, or projects inputs into a lower dimension; (2) learner robustification, which stabilises the training, applies moving target, or leverages ideas from robust statistics.
Countermeasures against attacks on RL models adopt similar approaches. For example, Mandlekar et al. [15], Pattanaik et al. [25] propose different adversarial training algorithms. Based on the idea that adversarial samples are not effective for the frame prediction module, Lin et al. [13] use previous images to predict future input and detect adversarial examples. Havens [9] propose the Meta-Learned Advantage Hierarchy (MLAH) framework that estimates advantage to measure the underlying changes in a task, in order to detect the attack.

Conclusions and Future Work
In this paper, we show that in the context of autonomous defence in cyber networks, RL agents can be manipulated by attacks that target the training process, even if the attacker only has partial observability of the environment and defensive algorithms. In order to defend against the attack, we propose an inversion method that aims to revert the perturbations added by the attacker. Our experimental results demonstrate the effectiveness of the proposed approach, and show that it causes limited impact in non-attack scenarios. Our work focuses on learning in software-defined networking, which brings with it novel threat models of independent interest to adversarial learning research.
For future work, we plan to (1) impose partial observability also on the defender, since in real networks, the defender may not be able to obtain the states of all the nodes all the time or the correct state of the nodes; (2) consider an attacker that can expand their partial observability as the attack proceeds; (3) consider an attacker that can spread more freely through the network, instead of having to compromise all the nodes on the paths to the critical server; and (4) replace the binary state with a continuous state, e.g., consider using multiple network performance metrics.