Leveraging AI to optimize website structure discovery during Penetration
  Testing
  Dirbusting is a technique used to brute force directories and file names on
web servers while monitoring HTTP responses, in order to enumerate server
contents. Such a technique uses lists of common words to discover the hidden
structure of the target website. Dirbusting typically relies on response codes
as discovery conditions to find new pages. It is widely used in web application
penetration testing, an activity that allows companies to detect websites
vulnerabilities. Dirbusting techniques are both time and resource consuming and
innovative approaches have never been explored in this field. We hence propose
an advanced technique to optimize the dirbusting process by leveraging
Artificial Intelligence. More specifically, we use semantic clustering
techniques in order to organize wordlist items in different groups according to
their semantic meaning. The created clusters are used in an ad-hoc implemented
next-word intelligent strategy. This paper demonstrates that the usage of
clustering techniques outperforms the commonly used brute force methods.
Performance is evaluated by testing eight different web applications. Results
show a performance increase that is up to 50% for each of the conducted
experiments.
