Behaviour-Diverse Automatic Penetration Testing: A Curiosity-Driven
  Multi-Objective Deep Reinforcement Learning Approach
  Penetration Testing plays a critical role in evaluating the security of a
target network by emulating real active adversaries. Deep Reinforcement
Learning (RL) is seen as a promising solution to automating the process of
penetration tests by reducing human effort and improving reliability. Existing
RL solutions focus on finding a specific attack path to impact the target
hosts. However, in reality, a diverse range of attack variations are needed to
provide comprehensive assessments of the target network's security level.
Hence, the attack agents must consider multiple objectives when penetrating the
network. Nevertheless, this challenge is not adequately addressed in the
existing literature. To this end, we formulate the automatic penetration
testing in the Multi-Objective Reinforcement Learning (MORL) framework and
propose a Chebyshev decomposition critic to find diverse adversary strategies
that balance different objectives in the penetration test. Additionally, the
number of available actions increases with the agent consistently probing the
target network, making the training process intractable in many practical
situations. Thus, we introduce a coverage-based masking mechanism that reduces
attention on previously selected actions to help the agent adapt to future
exploration. Experimental evaluation on a range of scenarios demonstrates the
superiority of our proposed approach when compared to adapted algorithms in
terms of multi-objective learning and performance efficiency.
