{'fraction_non_alphanumeric': 0.028440122552284536, 'fraction_numerical': 0.0044958039163447445, 'mean_word_length': 4.78036573628489, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Active Deception using Factored Interactive POMDPs to Recognize Cyber Attacker's Intent

This paper presents an intelligent and adaptive agent that employs deception to recognize a cyber adversary's intent. Unlike previous approaches to cyber deception, which mainly focus on delaying or confusing the attackers, we focus on engaging with them to learn their intent. We model cyber deception as a sequential decision-making problem in a two-agent context. We introduce factored finitely nested interactive POMDPs (I-POMDPx) and use this framework to model the problem with multiple attacker types. Our approach models cyber attacks on a single honeypot host across multiple phases from the attacker's initial entry to reaching its adversarial objective. The defending I-POMDPx-based agent uses decoys to engage with the attacker at multiple phases to form increasingly accurate predictions of the attacker's behavior and intent. The use of I-POMDPs also enables us to model the adversary's mental state and investigate how deception affects their beliefs. Our experiments in both simulation and on a real host show that the I-POMDPx-based agent performs significantly better at intent recognition than commonly used deception strategies on honeypots.


Introduction
An important augmentation of conventional cyber defense utilizes deception-based cyber defense strategies [16]. These are typically based on the use of decoy systems called honeypots [21] with additional monitoring capabilities. Currently, honeypots tend to be passive systems with the purpose of consuming the attacker's CPU cycles and time, and possibly logging the attacker's actions. However, the information inferred about the attackers' precise intent and capability is usually minimal.
On the other hand, honeypots equipped with fine-grained logging abilities offer an opportunity to better understand attackers' intent and capabilities. We may achieve this by engaging and manipulating the attacker to perform actions that reveal his or her true intent. One way of accomplishing this is to employ active deception. Active strategies entail adaptive deception which seeks to influence the attackers' beliefs and manipulates the attackers into performing desired actions [11]. We investigate how multi-agent decision making can be used toward automating adaptive deception strategies to better understand the attacker.
We represent cyber deception on a single host as a decision-making problem between a defender and an attacker. We introduce a factored variant of the well-known interactive partially observable Markov decision process [9], labeled as I-POMDP X , to computationally model the decision making of the defender while reasoning about the attacker's beliefs and capabilities as it acts and observes. I-POMDP X exploits the factored structure of the problem, representing the dynamics and observation function using algebraic decision diagrams, and solving the model using a method that directly operates on these factored representations [1]. This brings some level of tractability to an otherwise intractable framework, sufficient to adequately solve the cyber deception domain. I-POMDP X explicitly models the beliefs of the attacker and the defender throughout the interaction. This allows for detailed inferences about how specific deceptive actions affect the attacker's subjective view of the system. We evaluate the performance of I-POMDP X in promoting active deception with multiple attacker types both in simulation and on a real host. Our results show that the I-POMDP-based agent learns the intent of the attacker much more accurately compared to baselines that do not engage the attacker or immediately deploy all decoys en masse.

Background on I-POMDPs
Interactive POMDPs (I-POMDPs) are a generalization of POMDPs to sequential decision-making in multi-agent environments [9,4]. Formally, an I-POMDP for agent i in an environment with one other agent j is defined as, IS i denotes the interactive state space. This includes the physical state S as well as models of the other agent M j , which may be intentional or subintentional [3]. In this paper, we ascribe intentional models to the other agent as they model the other agent's beliefs and capabilities as a rational agent. A = A i × A j is the set of joint actions of both agents. T i represents the transition function, T i : The transition function is defined over the physical states and excludes the other agent's models. This is a consequence of the model non-manipulability assumption -an agent's actions do not directly influence the other agent's models. Ω i is the set of agent i's observations. O is the observation function, The observation function is defined over the physical state space only as a consequence of the model non-observability assumption -other's model parameters may not be observed directly. R i defines the reward function for agent i, R i : S i × A − → R. The reward function for I-POMDPs usually assigns utilities to the other agent's physical states.
We limit our attention to a finitely nested I-POMDP, in which the interactive state space IS i,l at strategy level l is defined bottom up as, Above,θ j represents agent j's frame, defined asθ j = A j , Ω j , T j , O j , R j , OC j . Here, OC j represents j's optimality criterion and the other terms are as defined previously. Θ j is the set of agent j's intentional models, defined as θ j = b j ,θ j . The interactive state space is typically restricted to a finite set of j's models, which are updated after every interaction to account for the belief update of agent j. The interactive state space for agent i at level l can be then defined as, Here, Reach(Θ j,l−1 , H) is the set of level l − 1 models that j could have in H steps; Reach (Θ j,l−1 , 0) = Θ j,l−1 . We obtain Reach() by repeatedly updating j's beliefs in the models in Θ j,l−1 .

Modeling Cyber Deception using Factored I-POMDPs
Engaging and deceiving human attackers into intruding controlled systems and accessing obfuscated data offers a proactive approach to computer and information security. It wastes attacker resources and potentially misleads the attacker. Importantly, it offers an untapped opportunity to understand attackers' beliefs, capabilities, and preferences and how they evolve by sifting the detailed activity logs. Identifying these mental and physical states not only informs the defender about the attacker's intent, but also guides new ways of deceiving the attacker. In this section, we first introduce our domain of cyber deception and subsequently discuss how it can be modeled in a factored I-POMDP.

Cyber Deception Domain
The cyber deception domain models the interactions between the attacker and the defender on a single honeypot host system. A state of the interaction is modeled using 11 state variables defining a total of 4,608 states. Table 1 briefly summarizes the state space. The S_DATA_DECOYS and C_DATA_DECOYS state variables represent the presence of sensitive data decoys and critical data decoys. The HOST_HAS_DATA variable represents the true type of valuable data on the system. We assume that a system cannot have two different types of valuable data simultaneously. This is a reasonable assumption because usually different hosts on enterprise networks possess different assets. We differentiate between sensitive_data and critical_data as distinct targets. Sensitive data, for example, includes private data of employees, high ranking officials, or any data that the attacker would profit from stealing. Also, in practical scenarios, honeypots never contain any real valuable data. Consequently, in the cyber deception domain in this paper, the HOST_HAS_DATA is none. However, the attacker is unaware of the honeypot or the data decoys and hence forms a belief over this state variable. Thus, the HOST_HAS_DATA variable gives a subjective view of the attacker being deceived. There are 5 observation variables for the attacker which make a total of 48 unique observations. We include three different types of attackers; the data exfil attacker, data manipulator and persistent threat. The data exfil attacker represents a threat that aims to steal valuable private data from the host. The data manipulator attacker represents a threat that seeks to manipulate data that is critical for the operation of a business or a physical target. Thus, the data exfil attacker targets sensitive_data in the system and the data manipulator attacker targets critical_data. The persistent threat attacker wants to establish a strong presence in the system at a high privilege level.
The attacker in the interaction can perform one of 9 actions to gather information about the system, manipulate the system, or take action on objectives. Table 2 briefly summarizes the actions available to the attacker. The FILE_RECON_SDATA and FILE_RECON_CDATA actions cause the DATA_FOUND variable to transition to yes. The FILE_RECON_SDATA action is slightly worse at finding data than the FILE_RECON_CDATA. This reflects the fact that private sensitive information is slightly difficult to find because it is often stored in user directories in arbitrary locations. On the other hand, critical data, like service configuration or database files, are stored in well-known locations on the system. The attacker gets information about the DATA_FOUND transition through the DATA observation variable. It simulates the data discovery phase of an attack. VULN_RECON is another action that works similarly and causes the VULN_FOUND transition to yes. This transition depicts the attacker looking for vulnerabilities to raise privileges. Depending on the type of the attacker, the START_EXFIL, MANIPULATE_DATA, or PERSIST actions can be performed to achieve the attacker's main objectives. We assume that the attacker is unable to discern between decoy data and real data, and hence, unable to determine which variable influences the DATA_FOUND state transition during file discovery. The attacker, however, can As DATA and DISCREPANCY are separate observation variables, the attacker can observe a discrepancy even when data has been found. When this occurs, the attacker develops a belief over the decoy data states as the host can have only one type of data. This realistically models a situation in which the attacker encounters multiple decoys of different types and suspects deception.
The defender in the interaction starts with complete information about the system. The defender's actions mostly govern the deployment and removal of different types of decoys. These actions influence the S_DATA_DECOYS and C_DATA_DECOYS states. Additionally, the defender can influence the attacker's observations about his privileges through the PRIVS_DECEPTION state. The defender gets perfect observations whenever the attacker interacts with a decoy. Additionally, the defender gets stochastic observations about the attacker's actions through the LOG_INFERENCE observation variable. The attacker is rewarded for exiting the system after causing an impact. For the data exfil and data manipulator attacker types, this is achieved by performing the START_EXFIL and MANIPULATE_DATA actions respectively. The persistent threat attacker is rewarded for getting root level persistence in the system. If decoys are indistinguishable from real data, the attacker attributes his observation to the existence of real data even when the host has none. Figure 1 illustrates a scenario taken from an actual simulation run with the data manipulator attacker type. Initially, the attacker has a non-zero belief over the existence of data on the system. However, the true state of the system on the left shows that the system does not actually contain any data. In the absence of the defender or any static data decoys, the attacker will eventually update his beliefs to accurately reflect the reality by performing the FILE_RECON_CDATA action and observing the result. However, to avoid this belief state, the defender deploys data decoys when the attacker acts. The attacker's inability to tell the difference between decoy data and real data and his prior belief about the absence of decoys leads him to attribute his observations to the existence of real data leading to the attacker being deceived.
(a) Dynamics compactly represented as a two time-slice DBN for select joint actions and observation variables.

Factored I-POMDPs for Modeling Cyber Deception
Factored POMDPs have been effective toward solving structured problems with large state and observation spaces [7,17]. Motivated by this observation, we extend the finitely-nested I-POMDP reviewed in Section 2 to its factored representation, I-POMDP X . Formally, this extension is defined as: the factored interactive state space consisting of physical state factors X and agent j's models M j . In a finitely-nested I-POMDP X the set M j is bounded similarly to finitely-nested I-POMDPs. Action set A is defined exactly as before. We use algebraic decision diagrams (ADDs) [1] to represent the factors for agent i's transition, observation, and reward functions compactly. T i defines the transition function represented using ADDs as P ai,aj (X |X ) for a i ∈ A i and a j ∈ A j . Y i is the set of observation variables which make up the observation space. O i is the observation function represented as ADDs, P ai,aj (Y i |X ). R i defines the reward function for agent i. The reward function is also represented as an ADD, R ai,aj (X ).
We illustrate I-POMDP X by modeling the cyber deception domain of Section 3.1 in the framework. Figure 2a shows the DBN for select state and observation variables given that the attacker engages in reconnaissance actions. The two slices in the DBN represent the sets of pre-and post-action state variables, X = {X 1 , ..., X n } and X = {X 1 , ..., X n } where X n represents a single state variable. Similarly, Y i = {Y i1 , ..., Y in } and Y j = {Y j1 , ..., Y jn } denote the sets of observation variables for agents i and j respectively. The ADD P ai (X |X , A j ) = P ai (X 1 |X 2 , . . . , X n , X , A j )×...×P ai (X n |X , A j ) represents the complete transition function for action A i = a i . This is analogous to the complete action diagram defined by Hoey et al. [13] for MDPs. Similarly, the observation function is represented using the ADD (Fig. 2b) which is analogous to the complete observation diagram [7]. Additionally, in an I-POMDP X , agent i also recursively updates the beliefs of agent j. The attacker types are modeled as frames in M j . Let M j = {m j1 : b j1 ,θ j1 , ..., m jn : b jq ,θ jr } be the set of all models in Reach(Θ j,l−1 , H). Because neither a j nor o j are directly accessible to agent i, they are represented as ADDs P (A j |M j ) and P ai (Y j |X , A j ). The distribution over M j is then Using these factors, we can now define the distribution over X and M j given action a i and observation o i as a single ADD using existential abstraction: Here, the ADD P ai (X |X , A j ) compactly represents T i (s t−1 , a t−1 i , a t−1 j , s t ), P ai (Y i |X , A j ) represents the probabilities O i (s t , a t−1 i , a t−1 j , o t i ), P (A j |M j ) represents P (a t−1 j |θ t−1 j ), and P ai (M j |M j , A j , Y j , X ) represents the recursive belief update transitions of the original I-POMDP. Thus, the constructed ADD P ai,oi (X , M j |X M j ) contains the transition probabilities for all interactive state variables given action a i and observation o i . The I-POMDP X belief update can then be computed as: where the ADD P ai,oi (X , M j |X , M j ) is obtained as in Eq. 1.
Symbolic Perseus [17] offers a relatively scalable point-based approximation technique that exploits the ADD structure of factored POMDPs. Toward generalizing this technique for I-POMDP X , we are aided by the existence of point-based value iteration for I-POMDPs [5]. Subsequently, we may generalize the α-vectors and its backup from the latter to the factored representation of I-POMDP X : Here, V t+1 is the set of α-vectors from the next time step and b i is a belief point from the set of considered beliefs B i . A popular way of building B i is to project an initial set of beliefs points forwards for H time steps using the belief update of Eq. 2.

Experiments and Analysis
We modeled the full cyber deception domain described in Section 3.1 from the perspective of a level-1 defender using the I-POMDP X framework. We implemented the generalized Symbolic Perseus using the point-based updates of the α-vectors and the belief set projection as given in Section 3.2, in order to solve I-POMDP X . The solver has several enhancements such as cached ADD computations and ADD approximations for additional speed up.
We evaluate the deception policy generated by I-POMDP X in simulations and on an actual system consisting of a standalone attacker programmed via Metasploit [15] and a defender workstation. We simulate each attacker type using the optimal policy computed by the level-0 attacker POMDP. We show these policies for each type of attacker in the supplementary material. For the simulations, we randomly sample the frame and the starting privileges of the attacker to simulate a threat with unknown intentions and privileges. The defender begins knowing about the existence of decoys on the system. The attacker, on the other hand, does not have prior knowledge about any vulnerabilities or data on the system. The defender engages with the attacker by deploying decoys, facilitating deceptive observations, or adding known vulnerabilities to the system. In the simulations, the state transitions and observations for both agents are generated by sampling from the joint transition functions and individual observation functions.
Simulations We compare the I-POMDP X policy against other passive baselines: one that does not engage and passively observes the attacker, and another which uses deception indiscriminately having deployed both sensitive and critical data decoys and all vulnerabilities in the honeypot at the beginning. We label the first baseline as NO-OP(no decoy) and the second baseline as NO-OP(all decoys). We perform the simulations for 30 trials with an attacker type randomly picked in each trial. During each trial, the defender begins not knowing the type of the attacker and believes that the state is that the attacker's privileges are not known. We set H in Reach(Θ j,l−1 , H) to 5. The generalized Symbolic Perseus is then run on 200 projected belief points until convergence to obtain the policy, which prescribes the subsequent actions for the defender until the end of the trial. It converges in about 6 minutes with a mean time per backup of 37 secs on Ubuntu 18 with Intel i7 and 64 GB RAM.
The NO-OP(no decoy) and NO-OP(all decoy) yielded a mean (± std err.) of 4.30 ± 0.16 and 3.26 ± 0.20 steps of engagement with the attacker, respectively. The longest engagement among these consisted of 7 and 5 steps, respectively. With NO-OP(no decoy), the attacker spends time searching for data and attempting to escalate his privileges but without much success, finally exiting the system. With NO-OP(all decoys), the attacker either quickly exploits the vulnerabilities or encounters the data decoys but quickly exits often due to the encountered data not being as expected. However, the I-POMDP X agent engaged with the attacker for a mean duration of 5.90 ± 0.24 with the longest interaction happening for 9 steps. It leverages the information gained by the first few observations to avoid using decoys that the attacker would find suspicious. For example, the defender first manipulates the attacker's observations about his privileges. This increases her chances of observing file enumeration or vulnerability discovery activity, forming a belief over the frames. Subsequently, the defender baits the attacker using decoys and observes the interaction to solidify her belief. This minimizes the risk of the attacker encountering unexpected decoys or noticing discrepancies.
The simulations above are predicated on the level-1 defender believing that none of the level-0 attacker types are aware of the deception, which is the typical case. However, if the defender believes that the attacker believes that there is a small chance at 0.1 of decoys being used, we observed that the attacker often quickly exited the system as one would expect. Do the extended engagements facilitated by the I-POMDP X agent help in intent recognition? Figure 3a(a) shows the cross-entropy between the defender's belief of the attacker's frame and the attacker's true type, as it varies across the steps of the interaction. The defender's I-POMDP X policy eventually yields the lowest cross-entropy values compared to the baselines, often reaching zero in 6 steps. We show the cross-entropy for more steps because the attacker remains in the system performing a few more actions. The sharp decrease in cross-entropy in the first three steps is because the attacker's decoy interactions (if the attacker is of type data exfil or manipulator) are perfectly observed by the defender (some other interactions generate noisy observations). Multiple consecutive data reconnaissance steps filter out the persistence attacker type, and the final step of either exfiltrating the data or manipulating it allows the defender to distinguish between the remaining two attacker types. But, for the NO-OP(no decoy) with no deception, the only source of information about the attacker is his general actions, which is noisy. Hence, such a defender is unable to form accurate beliefs before the attacker leaves the interaction. For the NO-OP(all decoy) agent that indiscriminately uses deception, observations from decoy interactions are perfect, but the risk of the attacker encountering contradicting decoys and suspecting deception is also high leading to early exits.
Host deployment In our next phase of experimentation, we evaluated the real-world feasibility of deploying an operational I-POMDP X on a host system and testing its efficacy. The testbed consists of 3 separate hosts: the attacker, the adaptive honeypot and the defender. Figure 4 shows the overall architecture of our testbed implementation. The attacker system runs a Kali Linux distribution which is well known for the variety of offensive and defensive cybersecurity tools that are preinstalled on it. The adaptive honeypot on which the interaction takes place runs a Metasploitable 3 Linux distribution. This distribution has a wide range of builtin vulnerabilities and is commonly used to simulate victim workstations in cyber attack simulations. The adaptive honeypot also contains an attacker agent that executes the attacks and communicates with the attacker. The attacker agent implements the actions given by the attacker's optimal plan located on the attacker host using realistic techniques commonly used by real attackers. We implement real exploits to facilitate privilege escalation on the host. The adaptive honeypot also has a defender agent that implements the defender's actions and gets observations.
The defender AI located on the defender workstation solves the I-POMDP X and computes the optimal action. For implementing the observation function, the I-POMDP X agent monitors and analyzes the system logs to get information about the attacker's actions (i.e., observations). To enable this, we use GrAALF [19], a graphical framework for processing and querying system call logs. GrAALF analyzes system call logs in real-time and provides the stochastic LOG_INFERENCE observation variable values (pertaining to file and vulnerability searches) as well as the perfectly observed DATA_DECOY_INTERACTION variable values to the defender.
Our results in Fig. 3(b) show the adaptive deception strategy employed by the I-POMDP X agent is better at engaging adversaries on a honeypot as compared to the passive strategies that are commonly used. While the cross entropy does not reach zero due to the challenge of accurately inferring the attacker's actions from the logs (leading to noisier observations), it gets close to zero, which is indicative of accurate intent recognition.

Related Work
AI methods are beginning to be explored for use in cyber deception. An area of significant recent interest has been game-theoretic multi-agent modeling of cyber deception, which contrasts with the decision-theoretic modeling adopted in this paper.
Schlenker et al. [18] introduced cyber deception games based on Stackelberg games [20]. These model deception during the network reconnaissance phase when the attacker is deceived into intruding a honeypot. Another similar approach [6] allocates honeypots in a network using a Stackelberg game. The game uses attack graphs to model the attacker and creates an optimal honeypot allocation strategy to lure attackers. Jajodia et al. [12] develop probabilistic logic to model deception during network scanning. While these efforts focus on static deployment of deception strategies at the network level, we seek active deception at the host level -once the attacker has entered the honeypot. Further, we model individual phases of the attack in greater detail, which allows us to employ realistic deception techniques at each phase.
At the host level, Carroll et al. [2] models deception as a signaling game while Horak et al. [10] creates a model for active deception using partially observable stochastic games. However, both of these take a high-level view modeling defender actions rather abstractly. In contrast, our defender actions are realistic and can be implemented on honeypots as demonstrated in Section 4. Ferguson-Walter et al. [8] model possible differences between the attacker's and defender's perceptions toward the interaction by modeling cyber deception as a hypergame [14]. Hypergames model different views of the game being played from the perspective of the players. While this approach similar to ours represents the attacker's perspective of the game, we explicitly model the adversary using a subjective decision-theoretic approach and do not solve for equilibrium.

Conclusion
Our approach of utilizing automated decision making for deception to recognize attacker intent is a novel application of AI and decision making in cyber security. It elevates the extant security methods from anomaly and threat detection to intent recognition. We introduced a factored variant of the well-known I-POMDP framework, which exploits the environment structure and utilized it to model the new cyber deception domain. Our experiments revealed that the I-POMDP X -based agent succeeds in engaging various types of attackers for a longer duration than passive honeypot strategies, which facilities intent recognition. Importantly, the agent is practical on a real system with logging capabilities paving the way for its deployment in actual honeypots.

Broader Impact
On a broader scale, the I-POMDP X framework that we introduce makes I-POMDPs tractable to be applied to larger problems. I-POMDPs are suitable for modeling multi-agent interactions due to their ability to model opponents from the perspective of an individual. This has a multitude of applications like negotiations, studying human behavior, cognition, etc. Through our work, we hope to make I-POMDPs tractable to be applied to such domains. Another area that we hope to motivate through our research is deception in human interactions. Modeling other agents explicitly will help understand how deceptive or real information influences an individual's beliefs. This has a wide range of potential applications such as studying how biases can be exploited, the effect of fake news on individuals, and how individuals can detect deception. We hope our research will eventually motivate further research in areas like counter deception and deception resilience in agents.
At an application level, our work aims to motivate the use of AI and decision making to create informed cyber defense strategies. Our work provides a new perspective different from the traditional action-reaction dynamic that has defined interactions between cyber attackers and defenders for years. Our framework models the opponent's mental states and preferences. This will aid security teams in understanding threats at a deeper level. We hope our framework will motivate the development of adaptive and intelligent deceptive solutions that can study and predict attackers at a deeper level. Understanding attackers' mental models, inherent biases, and preferences will go a long way in forming flexible cyber defense strategies that can adapt to different threats.  The persistent threat type attacker aims to establish root level persistence on the host. Such attacks are common. Attackers establish a strong presence in an organization's network and stay dormant for an extended duration. For this attacker type, the policy consists of vulnerability discovery actions in the initial steps. The attacker escalates privileges by performing the PRIV_ESC action on finding vulnerabilities. Once the attacker has the required privileges, the PERSIST action is performed to complete the objective.
While all three attacker policies may seem significantly different from their actions, the defender's observations of these actions are noisy. The errors in observation come from the noisy nature of real-time log analysis. For example, the VULN_RECON action models vulnerability discovery on a host. This action involves looking through the local file system for any vulnerable scripts, enumerating system information, listing services, etc. Thus a VULN_RECON can be mistaken for a FILE_RECON_CDATA or a FILE_RECON_SDATA in real-time log analysis. Similarly, it is difficult to tell the difference between the FILE_RECON_CDATA and FILE_RECON_SDATA from logs alone. Hence, without baiting the attacker into performing further actions, it is challenging to infer the intent of the attacker from the first few actions.