RANK: AI-assisted End-to-End Architecture for Detecting Persistent
  Attacks in Enterprise Networks
  Advanced Persistent Threats (APTs) are sophisticated multi-step attacks,
planned and executed by skilled adversaries targeting modern government and
enterprise networks. Intrusion Detection Systems (IDSs) and User and Entity
Behavior Analytics (UEBA) are commonly employed to aid a security analyst in
the detection of APTs. The prolonged nature of APTs, combined with the granular
focus of UEBA and IDS, results in overwhelming the analyst with an increasingly
impractical number of alerts. Consequent to this abundance of data, and
together with the crucial importance of the problem as well as the high cost of
the skilled personnel involved, the problem of APT detection becomes a perfect
candidate for automation through Artificial Intelligence (AI). In this paper,
we provide, up to our knowledge, the first study and implementation of an
end-to-end AI-assisted architecture for detecting APTs -- RANK. The goal of the
system is not to replace the analyst, rather, it is to automate the complete
pipeline from data sources to a final set of incidents for analyst review. The
architecture is composed of four consecutive steps: 1) alert templating and
merging, 2) alert graph construction, 3) alert graph partitioning into
incidents, and 4) incident scoring and ordering. We evaluate our architecture
against the 2000 DARPA Intrusion Detection dataset, as well as a read-world
private dataset from a medium-scale enterprise. Extensive results are provided
showing a three order of magnitude reduction in the amount of data to be
reviewed by the analyst, innovative extraction of incidents and security-wise
scoring of extracted incidents.
