Modeling Penetration Testing with Reinforcement Learning Using
  Capture-the-Flag Challenges: Trade-offs between Model-free Learning and A
  Priori Knowledge
  Penetration testing is a security exercise aimed at assessing the security of
a system by simulating attacks against it. So far, penetration testing has been
carried out mainly by trained human attackers and its success critically
depended on the available expertise. Automating this practice constitutes a
non-trivial problem, as the range of actions that a human expert may attempts
against a system and the range of knowledge she relies on to take her decisions
are hard to capture. In this paper, we focus our attention on simplified
penetration testing problems expressed in the form of capture the flag hacking
challenges, and we analyze how model-free reinforcement learning algorithms may
help to solve them. In modeling these capture the flag competitions as
reinforcement learning problems we highlight that a specific challenge that
characterize penetration testing is the problem of discovering the structure of
the problem at hand. We then show how this challenge may be eased by relying on
different forms of prior knowledge that may be provided to the agent. In this
way we demonstrate how the feasibility of tackling penetration testing using
reinforcement learning may rest on a careful trade-off between model-free and
model-based algorithms. By using techniques to inject a priori knowledge, we
show it is possible to better direct the agent and restrict the space of its
exploration problem, thus achieving solutions more efficiently.
