{'fraction_non_alphanumeric': 0.03278411649902417, 'fraction_numerical': 0.010508932592703798, 'mean_word_length': 4.983494273523467, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 8, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Processing Tweets for Cybersecurity Threat Awareness

Receiving timely and relevant security information is crucial for maintaining a high-security level on an IT infrastructure. This information can be extracted from Open Source Intelligence published daily by users, security organisations, and researchers. In particular, Twitter has become an information hub for obtaining cutting-edge information about many subjects, including cybersecurity. This work proposes SYNAPSE, a Twitter-based streaming threat monitor that generates a continuously updated summary of the threat landscape related to a monitored infrastructure. Its tweet-processing pipeline is composed of filtering, feature extraction, binary classification, an innovative clustering strategy, and generation of Indicators of Compromise (IoCs). A quantitative evaluation considering all tweets from 80 accounts over more than 8 months (over 195.000 tweets), shows that our approach timely and successfully finds the majority of security-related tweets concerning an example IT infrastructure (true positive rate above 90%), incorrectly selects a small number of tweets as relevant (false positive rate under 10%), and summarises the results to very few IoCs per day. A qualitative evaluation of the IoCs generated by SYNAPSE demonstrates their relevance (based on the CVSS score and the availability of patches or exploits), and timeliness (based on threat disclosure dates from NVD).


Introduction
A security analyst must be aware of the latest developments regarding updates, patches, mitigation measures, vulnerabilities, attacks, and exploits to adequately protect an IT infrastructure. Security Operations Centers (SOC) improve their awareness through Security Information and Event Management (SIEM) software, thereby allowing the correlation of the latest cybersecurity developments with internal infrastructure events.
There are two primary ways of obtaining cybersecurity news. One is to purchase a curated feed from a specialised company such as SenseCy [4] or SurfWatch [14]. Another, is to collect Open Source Intelligence (OSINT) [43] available from various sources on the internet (e.g., Threatpost [15]).
There are numerous threat intelligence tools (e.g., Spider-Foot [13], IntelMQ [7]) that can collect security-related OS-INT from a wide variety of sources, including such feeds. However, these use simple keyword-based filters to narrow the big volume of collected information, not employing any sophisticated methodology to select only the relevant data or handling data aggregation and duplicate removal-two fundamental characteristics for an efficient OSINT usage [40]. Moreover, recent work (e.g., [29,37,50]) demonstrates that different types of useful information and Indicators of Compromise (IoC) can be obtained from OSINT through the application of machine learning techniques. These results highlight the gap between the current capabilities of existing OSINTprocessing tools and the intelligence OSINT can provide.
To address this gap, this paper proposes SYNAPSE, a Twitter-based streaming threat monitor that generates a continuously updated summary of the threat landscape concerning a monitored infrastructure. SYNAPSE gathers tweets from carefully selected security-related accounts, selects those relevant for the specified monitored infrastructure using supervised machine learning, and avoids presenting repeated information by employing a novel stream clustering method.
SYNAPSE's design addresses three main challenges: OS-INT collection, cybersecurity-related content selection, and the aggregation of related tweets through a stream clustering algorithm adapted to the context of cybersecurity. A threat intelligence tool must address the first two challenges to ensure its usefulness, i.e., it must collect large amounts of data and accurately select those that are relevant for the SOC. The aggregation challenge is paramount to promote the efficient operation of the SOC, i.e., the analysts have a limited time budget to evaluate the current threat landscape, thus the tool must present only a summary of the most relevant information. This summarised view enables prioritising threats that require exploring additional information such as the tweets' links.
Twitter was chosen for two main reasons. First, Twitter is well-recognised as a relevant source of short notices (almost in real-time) about web activity and occurring events [5]. Previous research shows this is also true for cybersecu-rity [19,32,37]. In fact, the most important cybersecurity news feeds are present in Twitter (e.g., NVD, ExploitDB, CVE, Security Focus, Nessus), making it a hub for all these sources. Second, the limited size of a tweet makes it simple to process through general-purpose machine learning approaches, which enable low error levels across multiple domains of application. Furthermore, although short, tweets provide enough elements to categorise their content, as well as links for more detailed material.
Most previous work to gather cybersecurity OSINT information focuses on the filtering and classification process [27,31,33,34,37]. Beyond that, few works extract information from unstructured text (including tweets) [21,29,45,50]. However, to the best of our knowledge, no previous work addresses the timely summarisation of a cyberthreat Twitter stream, thereby providing an end-to-end approach for monitoring the current threat landscape.
The standard technique for aggregating related data is clustering [16]. In a streaming context, a stream clustering algorithm becomes necessary [42]. Existing algorithms (e.g., [24,48,49]) have two shortcomings for our context: they require a priori definition of the target number of clusters and they discard outliers. However, when processing a cybersecurity news feed, the number of active threats under discussion is unknown in advance and outliers cannot be discarded as they are likely to represent new threats. SYNAPSE adapts a well-known stream clustering algorithm to overcome these limitations, by detecting whether tweets refer new threats or updates to previously known ones, thus becoming appropriate for maintaining a continuous upto-date summary of current threats observed. Finally, to close the pipeline from Twitter to the SOC tools, SYNAPSE produces IoCs from the clustered OSINT, making it integrable with various SIEMs (e.g., IBM QRadar [6]) and threat intelligence/sharing tools (e.g., MISP [10]).
A quantitative evaluation considering all tweets from 80 accounts over more than 8 months (over 195.000 tweets), shows that SYNAPSE finds the majority of security-related tweets concerning an example IT infrastructure (true positive rate above 90%), incorrectly selects a small number of tweets as relevant (false positive rate under 10%), and summarises the results to very few IoCs per day. When compared to a naive text-filtering approach (as employed by most threat intelligence systems used in practice), it decreases the number of tweets presented by approximately 80%, with the number of summarised IoCs being only 21% of the tweets classified as relevant. This volume of data can either be inspected manually or processed by a SIEM as OSINT-generated events. Further, a qualitative analysis of the largest 65 clusters generated by SYNAPSE revealed two paramount findings. Firstly, 43% of the IoCs describe high-impact security alerts (CVSS ≥ 7.0), and for half of these, the tweet publication preceded the vulnerability publication on the National Vulnerability Database (NVD) by eight days (on average). Secondly, 70% of the analysed clusters provided serviceable intelligence, including exploits whose vulnerabilities were not matched to NVD entries. In summary, our contributions are: 1. An end-to-end streaming threat monitor architecture for collecting, classifying, and clustering tweets related to a specified infrastructure (Section 3); 2. A novel application strategy and adaptation of wellknown clustering techniques to the context of cybersecurity threat awareness (Section 4); 3. A detailed system evaluation using three real-world datasets and a qualitative analysis of the security alerts generated thereof (Section 7); 4. Methods for generating MISP-compatible IoCs from tweets that enable the integration of SYNAPSE into SOC operation (Sections 3 and 5).

Related Work
In the following, we briefly review the previous work related to SYNAPSE: processing tweets for cybersecurity, threat intelligence tools, and stream clustering algorithms.
Twitter for cybersecurity. Several works aim to find cybersecurity OSINT about a given IT infrastructure. These rely on a keyword set to govern the selection of tweets, thereby picking only the potentially relevant content. Mittal et al. [33] use a knowledge base created from security concepts to evaluate if a tweet is relevant for cybersecurity. Similarly, Le Sceller et al. [27] designed a framework that collects tweets on a keyword basis and is capable of extending the keyword set automatically. Ritter et al. [34] search Twitter for occurrences of three specific topics: DoS attacks, data breaches, and account hijacking. Trabelsi et al. [45] cluster tweets by subject. Threats not referred by NVD are considered novel and handled like zero-day vulnerabilities. Dionísio et al. [21] used deep learning techniques to detect and extract security-related information from tweets. Sabottke et al. [37] show that information about exploits are published on Twitter two days before they are included in NVD (on average). None of these works provide an end-to-end solution for online threat monitoring, mainly because they focus on detection, overlooking summarisation and SOC integration. IoC generation Figure 1: SYNAPSE's architecture. Collected tweets pass through the various stages and those classified as relevant are aggregated, transformed in IoCs, and delivered to SOC analysts. literature, enabling high recall of the methodology. In a different work, Zhu et al. [50] present a system that processes the scientific literature studying Android malware and extracts features describing the attacks to create a malware detector. The objective of these works is to extract machine-readable information from OSINT, which is different from our goal.
Besides the research-oriented efforts to include OSINT in protection systems, off-the-shelf tools are able to collect and deliver OSINT-based threat intelligence. SpiderFoot [13] is an OSINT automation tool that uses multiple sources (e.g., Bitcoin addresses, Twitter) for three main purposes: target reconnaissance, assess an organisation's exposure on the Internet, and OSINT collection for security purposes. IntelMQ [7] is an open-source system for collecting and processing securityrelated OSINT feeds designed for organising data coming from various sources. It employs an ontology for data harmonisation and converts all events into a uniform json format. MISP [10] is a threat intelligence platform designed for sharing and correlating IoCs. It receives many types of threat inputs and exports its data into other MISP instances or threat intelligence tools. Generally speaking, these tools do not employ any advanced processing capability for filtering and matching threats, resorting only to keyword-based string comparisons.
Stream clustering. With the few exceptions discussed bellow, most stream clustering algorithms require the target number of clusters (k) to be defined as a parameter and discard elements that do not fit the clusters (outliers) [42]. Feng et al. [22] cluster only the tweets' hashtags, using text similarity to adapt the number of clusters to the collected data. However, this algorithm would potentially miss important information in the security field, as the clustering would not consider the full tweet text, only hashtags. Saki et al. [38] use a densitybased clustering approach, therefore avoiding the definition of k. However, their technique discards outliers, which could lead to missing important emerging threats. Shou et al. [41] algorithm allows the value of k to vary up to an upper limit, but its outlier detection mechanism discards topics that do not gain traction, ignoring possibly important threats that remain unknown for long periods of time. Figure 1 presents SYNAPSE's architecture and data processing stages-tweet gathering, filtering, feature extraction, classification, clustering, and IoC generation-described next.

SYNAPSE Pipeline
Data Collection. The data collector module requires a set of accounts, from which it will collect every posted tweet using Twitter's stream API-an approach already found in the literature [39]. These accounts can be from security analysts and organisations, vendors, hackers, researchers, among others. They are chosen considering the likelihood of users tweeting about the security of elements belonging to the monitored IT infrastructure. Since usually security analysts already follow OSINT sources and Twitter accounts, it is just a matter of providing these sources to SYNAPSE.
Simply collecting tweets by keywords is a method likely to retrieve large amounts of irrelevant information. For instance, tweets with the word "windows" include all Windows-related topics (the OS) and all tweets referring glass windows. By collecting tweets only from selected security-related accounts, a more substantial fraction of tweets is related to cybersecurity.
Filtering. Despite the account-based collection approach, most likely the collected data will include tweets unrelated to the infrastructure under the analyst's care. These have to be dropped by a filter. The filtering approach assumes that a tweet referring a threat to a particular IT infrastructure asset has to mention that asset. Therefore, a second input is required: a set of keywords describing the assets of the monitored IT infrastructure. Only tweets that include at least one of the keywords will pass the filter. Keywords further restrict the scope of the security events, hence decreasing the number of irrelevant tweets beyond the filter.
To maximise the effectiveness of SYNAPSE, the keywords defining the monitored assets must be as complete and specific as possible. For example, if the analyst is in charge of securing a Linux cluster running virtual machines to serve a web service with a database, the keyword set could be {linux, ssh, virtualbox, vbox, mysql, apache, php}.
Pre-processing and Feature Extraction Pre-processing normalises the tweet representation. First, all characters are converted to lower case, and stopwords and hyperlinks are removed-the latter are shortened URLs that provide little information. Numbers, dots, and hyphens are replaced by their textual representation (e.g., "2" to "two"), as these are relevant to distinguish software versions (e.g., Mozilla Firefox 4.5.1-2). Finally, all non [a-z] characters are removed. For instance, after pre-processing, the tweet "#Oracle #Linux 6 / 7 : Unbreakable Enterprise kernel (ELSA-2016-3573) https://t.co/vLTel8NodG" becomes "oracle linux six seven unbreakable enterprise kernel elsa hyphen two thousand and sixteen hyphen three thousand five hundred and seventy three". The original tweets are stored for presentation.
The tweets must be converted to a numerical format to become suitable for supervised learning classification techniques. This work uses the well-known Term Frequency -Inverse Document Frequency (TF-IDF) method [28]. TF-IDF computes weights to words (features) based on their occurrence frequency in each document and on the group of documents considered. The weight of a word increases with its frequency of occurrence in a single document but is scaled down by the frequency of occurrence in all documents. By mapping each consecutive word token to a corresponding vector position, tweets are converted to a constant size, zeropadded, TF-IDF numeric vector. Finally, to limit the size of the vector we employ the hashing trick technique [46].
Classification. For the classification of tweets according to their security relevance, two classifiers have been explored: Support Vector Machines (SVM) [20] and Multi-Layer Perceptron (MLP) Neural Networks (NN) [35,36]. The SVM is a broadly-used classifier achieving good results across a multitude of application domains. We consider the SVM implementation available in the Apache Spark' Machine Learning library (MLlib) [2], which employs a linear kernel, thereby assuming the input vectors are linearly separable.
Since MLlib does not provide a non-linear SVM kernel, MLlib's MLP NN implementation was considered to account for the assumption that input vectors may not be linearly separable. The MLP is a well-established and frequently used NN architecture that has a long track record of good and consistent results over a vast number of classification tasks.
Clustering. SYNAPSE uses clustering to aggregate similar tweets in the news feed stream. The Clustream algorithm [17] Figure 2: Representation of a cluster into the MISP taxonomy [11] and an OSINT-generated event in MISP.
was chosen as the basis for this pipeline stage as its structure and characteristics were closest to our requirements. However, it required adaptation to SYNAPSE's context to achieve threat aggregation as described in the next section.
MISP compatible IoC Generation. After the clustering phase, the clusters of tweets are transformed into the IoC format to allow their inclusion in SIEMs or threat intelligence platforms. There are several standards for sharing IoCs, such as STIX [8] or MISP [11]. The format must be extensible and adaptable as tweets are unstructured and contain unpredictable content. For these reasons, the MISP format has been selected to generate IoCs. Moreover, it can be easily converted into other standard formats like STIX. We use a combination of MISP items to generate the IoC. One MISP Event is composed of two Objects containing security indicators called Attributes: one describing the content of the exemplar tweet (the cluster centroid); the other representing the cluster of tweets. Events are classified using tags, added according to a set of threat categories related to existing taxonomies: ENISA and VERIS for cyberthreats [12]. The OSINT tag is added to emphasise the automatic creation based on tweets. The classification is achieved by using regular expressions to match taxonomy elements in the exemplar's message, generating one tag for each match. Figure 2 depicts the taxonomy employed to represent IoCs in MISP (top of the figure). The exemplar tweet is the core of the IoC, while its cluster is an extra element to increase informativeness. The bottom of the figure shows a MISP Event generated from a cluster and its exemplar (the example cluster shown in Table 1). The OSINT object contains extracted information from the exemplar such as the tweet's message, any links therein, and the Cluster_Analysis object contains the remainder cluster data. A simple classification was applied: Table 1: An example of a cluster and its exemplar (in Bold).
Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP POST Denial of Service Vulnerability https://t.co/6FXInr9hNh Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP POST Denial of Service Vulnerability https://t.co/6FXInr9hNh Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP Length Denial of Service Vulnerability https://t.co/TgU0T9vlZt #bugtraq Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP POST Denial of Service Vulnerability https://t.co/feZlTxQKVC #bugtraq #cybersecurity Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP POST Denial of Service https://t.co/XUUctUnQ8F #infosec #vulnerability #security : Bugtraq: Cisco Security Advisory: Cisco Web Security Appliance HTTP POST Denial of Serv https://t.co/9bW0ls00kx #internet #security: Cisco Web Security Appliance HTTP POST Denial of Service Vulnerability https://t.co/cXQUTWUBbD the OSINT tag marks the event as created from tweets, and the "Denial of Service" tag (from VERIS) classifies the threat.

Tweet stream clustering
Since Twitter users can tweet or retweet about the same subject, SYNAPSE is expected to collect many similar tweets. Thus, to cover information about the IT infrastructure, the analyst would have to manually inspect a large amount of redundant data for each threat.
To alleviate this burden, clustering is used to group similar tweets classified as relevant for the protection of the IT infrastructure. Ideally, the information collected about a specific threat gets aggregated in one cluster, from which a single representative tweet-the exemplar-is presented to the analyst. By clustering the stream of relevant tweets, distinct active threats are summarised in a set of clusters and updated as more tweets are collected. It is through this mechanism that SYNAPSE can create an active threat monitor outlining the current threat landscape, i.e., the current threats that potentially require more immediate attention from SOC analysts.

Data stream aggregation challenges
Clustering is commonly applied in batch, as an exploratory data technique where a static data set is clustered into k groups [16]. The number of clusters, k, is either defined a priori or estimated to satisfy performance metrics [16]. In a dynamic setting such as SYNAPSE's streaming context, defining k beforehand is not possible, as the number of threats being discussed at a given time is unknown. If at any moment SYNAPSE was processing t threats and clustering was set to find k = t clusters, the result would contain clusters including unrelated threats, various clusters related to the same threat, or both cases. Therefore, SYNAPSE requires a clustering algorithm able to adapt k over time.
Furthermore, an essential feature of most stream clustering algorithms is the ability to detect and remove outliers that may disrupt the quality of the clustering. In the security context, performing outlier removal could prevent the discovery of emerging threats. Moreover, all tweets reaching SYNAPSE's clustering stage were classified as relevant, and should not be discarded. Therefore, SYNAPSE requires a clustering algorithm capable of maintaining performance indicators (e.g., intra and inter-cluster cohesion) without removing outliers.

DynamicClustream
The lack of solutions that fit the requirements of threat intelligence tools (see Section 2), motivated us to adapt the Clustream [17] algorithm for SYNAPSE, thus creating the DynamicClustream. The Clustream algorithm clusters a data stream in two phases. The online phase performs a simple and efficient clustering of the inbound stream by keeping only a summary of the data collected, thus abiding to the speed requirements of a data stream [16]. The offline phase is performed in background to provide a more complete analysis of the collected data through a more effective and computationally demanding clustering algorithm. Clustream includes an outlier detection mechanism that excludes data points unfit for any of the existing clusters by analysing the distance from that point to all clusters. A decision is only taken once it becomes clear if a data point is an element of a new trend or an isolated occurrence. The components that distinguish Dynam-icClustream from Clustream are detailed in the following.
High-level Overview. Assume there is always a global cluster state S, defined as a set of sets, describing the clusters formed from a previously processed time-window of tweets. When a new tweet t is received, the online clustering component attempts to place t in one of S's clusters. If a direct placement is not possible, the offline clustering component is triggered to compute a new clean cluster state considering the tweets in the clusters of S plus t.
Once a new cluster state S is in place, a final step is taken to obtain each cluster's exemplar tweet, i.e., the tweet representing the cluster, that will be shown to the analyst. The exemplar tweet is selected by choosing the tweet with the smallest Euclidean distance to the centroid of the cluster. An example of a generated cluster (and its exemplar) appears in Table 1. The online and offline components of DynamicClustream are presented in Algorithm 1, with locking details for ensuring atomic updates on S omitted for better readability.
Online clustering component. The online clustering component uses a lightweight approach to assign a new tweet t to the current clustering state S. To do so, the membership of t is tested in all clusters (line 3) by employing the WTS cohesion measure (introduced below). This is done by adding t to each cluster C i ∈ S and calculating the corresponding WTS value. t belongs to C i when WTS is above a certain threshold τ. If t does not fit in one of the existing clusters, a new cluster solely containing t is created (lines 4-5). If t belongs to a single cluster, it is added to that cluster (lines 6-7). When t fits more than one cluster, it is added (temporarily) to the cluster PlaceInClosestCluster(S,t) 10 schedule OfflineClustering(S) 11 Function OfflineClustering(SavedState): with the highest membership rate, and the offline clustering is scheduled (lines 9-10).
In SYNAPSE's application scenario it makes no sense to remove outliers. Instead, when new tweets do not belong to S, we treat them as the onset of a threat by adding new clusters with a single element which in time may receive additional tweets. This outlier processing mechanism allows adapting the number of clusters, k, to the novelty in the dataflow. Furthermore, it is through the online component of Dynamic-Clustream that the active threat monitor is implemented: the system categorises new tweets as new threats or as updates to known ones, thus maintaining an updated threat summary about an IT infrastructure.
Cohesion Measure. Cluster cohesion and cluster separation are concepts used to assess the validity of a partition generated by a clustering algorithm [18], which in most cases have a purely geometric interpretation. In SYNAPSE, cohesion is based on the similarity of tweets within a cluster and not on a geometric measure such as the distance to the cluster centroid, thus defining a context-based cluster validation approach, argued to be more effective [25].
To reinforce the one-to-one relation between clusters and threats, the cohesion measure must detect clusters whose tweets refer to the same threat. Assuming that a threat is expressed by a minimum number of words appearing in all tweets, the proposed cohesion measure-named Withincluster Threat Similarity (WTS)-is defined as ω w m , where ω is the number of words shared by all the cluster's tweets and w m is the number of words of the smallest tweet in the cluster. WTS is 0 if no words are shared by the tweets of a cluster, and 1 when all tweets share the words of the smallest tweet in the cluster. It assumes that if all cluster tweets share a sufficiently large number of words, then they mention the same threat.
The degree of separation of two clusters C i and C j is measured by the Jaccard index [47]. It is determined as J = |C i ∩C j | |C i ∪C j | , corresponding to the ratio between the number of common words to C i and C j and the number of unique words of C i and C j . The lower its value, the more separated the clusters are.
Offline clustering component. The offline component applies the k-means clustering algorithm [30] repeatedly to provide more robust clusters. k-means is a widely used algorithm that has provided good efficiency and empirical success over the last 50 years [26]. However, it is commonly employed for exploratory data analysis, not for automatic text summarisation.
The k-means algorithm requires the specification of the number of clusters, k, which is unknown in this case. At a given time we do not know how many potential threats to our infrastructure are being discussed. Therefore, we defined a novel strategy to find the so-called elbow point [44], i.e., the point beyond which by increasing k there is no significant improvement in the clusters' Sum of Squared Errors (SSE). This procedure automatically determines k, thus avoiding the specification of a threshold to find the elbow point or the visual inspection of the within-class-variance versus k graph.
k-means application strategy: Starting at k 1 = 2, a k-means model is trained for each successive k i = i + 1 number of clusters, which produces a corresponding SSE, denoted by ε i . As the initial cluster centres are randomly chosen, there is a given variance σ i associated to ε i . As we keep increasing k i , we expect ε i to decrease up to the point where the magnitudes of ε i and σ i become of the same order. At this point ε i+1 − ε i might become zero or even negative, indicating that there is no significant SSE improvement in increasing k i . Therefore, the iteration is stopped when the error (ε) stops decreasing or (the limit case where) the number of clusters corresponds to the number of tweets to be clustered, and k i is selected as the number of clusters (lines [16][17][18][19][20][21]. By testing this approach, we found that small clusters had only very similar tweets, but other large clusters contained unrelated tweets. The cause might be two-fold: (1) k-means assumes spherical clusters that it tends to produce equally sized, which might not be adequate; and (2) the strategy to find k is not guaranteed to find the best k. To overcome this limitation, we use the WTS cohesion measure to quantify how closely related the tweets in a cluster are, and implement a re-clustering method that splits these clusters into smaller ones with related tweets. If WTS ≥ τ (a specified threshold), indicating high cohesion, it enables the validation of clusters as final.
Re-clustering method: All tweets of non-final clusters are gathered (line [22][23][24][25] and re-clustered (lines 16-21) using k-means to allow similar tweets to be grouped. Then, the new clusters generated are again tested using their WTS, and the process is repeated for the non-final clusters. Eventually, all clusters are considered final, ideally each related to a single threat, and S * is merged with S (line 26), i.e., S * is updated with the tweets received since the algorithm started by executing a procedure similar to lines 3-10.
Offline clustering scheduling: At any time, there may be only one instance of the offline component in execution. Since multiple tweets received in a short time interval may trigger offline clustering, we employ the schedule keyword (line 10) to avoid overlapping executions. The idea is that each call to schedule OfflineClustering() notifies the system that offline clustering is required after this point, and saves the current cluster state for its next execution. Once the algorithm is started again (using the latest saved state), it process all tweets pending in S (line 12).
Time-window model. To fully adapt Clustream to our context we also changed the clustering ageing model used to remove clusters. This model is necessary to complete the adaptation of the cluster state to the data stream flow.
Clustream's window model is global in the sense that all data points are aged and removed using the same rule. However, this methodology does not fit SYNAPSE application domain, as different cybersecurity topics have different lifetimes. For example, news about an update are expected to last a few days, while advances about an active threat may continue for a month or more. Thus, in the cybersecurity field it makes more sense to adopt a local window model, monitoring ageing by cluster (by threat). As a consequence, whole clusters rather than single points should be removed in forthcoming clustering states.
In DynamicClustream a cluster C i is removed from the cluster state S if it has been stale for a period of time longer than θ, i.e., if θ time passes without C i receiving a new data point. In this way, topics that no longer receive traction are stowed away, while active topics retain all their elements, regardless of the time passed, which may be crucial for understanding the evolution of a threat.

SOC integration
An essential aspect of threat intelligence tools such as SYNAPSE is the integration in a SOC. In the following, we describe practical issues related to this integration.
Twitter as OSINT. When using Twitter as a cybersecurity information source, it is important to consider what would happen if some of the monitored accounts fell under the control of the adversary. In a nutshell, two things can happen [37]: (1) the adversary may not tweet about the threats he is interested in exploiting using the accounts he controls; or (2) the adversary may create tweets with false threats, to make SOC analysts waste their time in solving potential non-existent problems. Both attacks should not be a significant problem as long as the amount of accounts controlled by the adversary is relatively small, and the analysts take into account the reputation of the accounts monitored by the system.
Training the system. Our approach requires the creation of labelled datasets for training the classifiers. To do that, the SOC analysts need first to configure the keywords defining the infrastructure. A second configuration step is to define the Twitter accounts that will be monitored.
After those two steps, the system should present all filtered tweets as if they are important, and a button for the analyst to mark a tweet as "irrelevant". 1 Notice that, to avoid bias, it is relevant to inform the analysts that the system is under training. When enough positively-labelled tweets are collected, the classifiers can be trained in background and then placed in operation.
It is expected that the classifier's performance decreases with time, as the operational data gets progressively different from the training data. To maintain the utility of the classifiers in use, it is essential to minimise this effect. Incremental learning is a technique that can be used for this purpose, where the classifier's model is continuously trained with new labelled examples [23]. By training the model with the latest events, it is continuously adapted to changes in input format (in this case, changes in tweet format or language).
Another possibility is to replace the model with a new model trained with only the latest data, e.g., the last three months of tweets. This way the model is periodically adapted to the current threat landscape, so that old data will not impact the classifier's quality.
Changing keywords and monitored accounts. Adding or removing keywords from the datasets require retraining the classifier. Removing a keyword requires removing the tweets that were filtered by this keyword and retrain the model without them. To add a keyword, one needs first to complement the existing labelled dataset (in the same way as described before) with tweets related to the new keyword, and then retrain the model with the reformulated data set. Changing the set of monitored Twitter accounts is not a burden for the system since the structure of threat descriptions is expected to be

Experimental Setup
This section describes the experimental work carried out to validate SYNAPSE. All code is written in Scala and deployed on the Apache Spark Framework [2]. We chose Spark as its data-structures are scalable and designed for large datasets. Also, Spark includes a scalable machine learning library called MLlib, used to implement all ML algorithms employed in this paper.
Infrastructure Definition. We used a hypothetical IT infrastructure to set SYNAPSE's filter during its experimental evaluation. This infrastructure (presented in Table 2) is composed of software elements typically found in the IT world, such as the most common browsers and operating systems.
Tweet Collection and Labelling. We collected three datasets during three periods of time. Table 3 presents their collection periods, the sets of accounts used, and the number of tweets. After being collected and filtered using the keywords in Table 2, each tweet was manually labelled as positive (the tweet mentions a threat to a given part of the IT infrastructure) or negative, thus creating labelled datasets suitable for supervised learning. Two sets of accounts, S1 and S2, were used for tweet collection, as shown in the third row of Table 3. The accounts are listed in Table 4.
Classifier Configuration. Supervised machine learning techniques require design tailored to the problem at hand. For each classifier employed, their relevant parameters and design variables were varied, namely the step size and the regularisation parameter (C) for the SVM, and the number of layers and neurons per layer for the MLP. The size of the TF-IDF feature vector considered was also varied for both classifiers. Through a Pareto-optimal search, ideal configurations were found: the  access, acl, admin, advisory, allow, arbitrary, aslr, assurance, attack, auth, buffer, bug, bypass, certificate, code, command, corruption, csrf, cve, cyber, denial, deployment, dereference, disclosure, execute, exploit, hack, heap, identity, injection, interception, leak, overflow, privilege, remote, root, scripting, security, stack, threat, unauthenticated, vuln, xss best SVM uses a step size and C of 0.05 and 5, respectively, and the best MLP had 5 layers with 10 neurons each. Both models use feature vectors with a size of 3000, revealing a clear advantage in using high-dimensional feature vectors. A complete description of the methodology employed for the classifier's design can be found in Appendix A.
Clustering. SYNAPSE uses the k-means algorithm in the offline clustering component, configured with fifty iterations, a minimum of two clusters, and the remaining parameters with their default values. Clustering was performed on the set of tweets classified as positive.
The WTS cluster cohesion measure was set to τ = 2 3 . This value was selected after preliminary experiments, reflecting the rationale that two tweets can be in the same cluster if and only if they share at least two-thirds of their words.
We compare our data presentation strategy with the one employed by threat intelligence tools and SIEMs capable of collecting OSINT (e.g., AlienVault OTX [1], Spiderfoot [13]). For that, we set up a Logstash [9] instance fed by the same dataset as SYNAPSE, which selected as relevant tweets mentioning at least one of our infrastructure assets and containing at least one security concept.
The security concept keywords were selected using the following methodology. First, a list of documents is obtained by selecting all tweets labelled as positive from all datasets. After that, we removed stopwords, applied the TF-IDF method, and selected the words with TF-IDF value lower than a threshold ρ. Finally, the list was manually filtered for security-irrelevant content (such as numbers). We considered ρ values of 0.1, 0.2, and 0.3. After inspecting the results, ρ = 0.2 was chosen due to the provision of the most substantial amount of generic words without showing words related to a specific context. The Logstash security concept keyword set corresponding to ρ = 0.2 appears in Table 5.
For the time-window model we applied a θ value of seven days, i.e., a cluster without updates for seven days is removed from the online clustering state. The same θ value was applied to the Logstash approach but globally, i.e., all relevant tweets were removed from the active threat pool after a week.

Results
The tweet processing pipeline components were evaluated using the selected models and datasets D2 and D3. These consider only tweets in the future of those in the training set (D1), and include information posted by an additional and substantially larger set of accounts (S2) not considered in the training stage. This methodology embodies the idea that in a real deployment, models will classify future tweets possibly from a different set of accounts.
Considering that 10-fold cross-validation was employed during the model selection phase, it should be noted that the selected model configurations were trained for the evaluation phase using the whole D1 dataset. The feature vectors of D2 and D3 tweets were generated using the TF-IDF model determined using dataset D1. This guarantees that TF-IDF weights attributed to words in D2 and D3 will be coherent with those used to train the classifiers. Figure 3 shows the True Positive Rate (TPR) and True Negative Rate (TNR) of the SVM and MLP classifiers described in Section 6, considering also the average result of the 10-fold cross-validation over D1.

Classification
Overall, the results are slightly worse for D2 and D3 when compared to D1 (as expected), since new data presents unmodeled patterns to the classifiers. Focusing on the results obtained for D2 and D3, in general, the classifiers maintain very high TPR and TNR, except for the MLP TPR. In both cases, the TNR is higher than the TPR. The imbalance between positively and negatively labelled data in the training data sets (more negative samples) can explain a higher TNR.
In summary, the SVM approach achieved the best results, displaying true positive and true negative rates around 90% and showing a small degradation of results in D2 and D3. For these reasons, the SVM model was employed in all further experiments. These results support the application of a supervised classifier to select tweets relevant for cybersecurity.

Clustering
The following experiments evaluate SYNAPSE's ability to aggregate the dataflow into meaningful clusters, where each cluster is expected to describe a single threat. Further, the DynamicClustream's window model is evaluated to assess its capability to detect the continuous discussion of threats. The initial clustering evaluation focuses on the basic algorithm's capability of properly aggregate tweets, i.e., producing clusters with high internal cohesion and low inter-cluster similarity. Then we analyse the end-to-end benefit of SYNAPSE and discuss the effectiveness of the proposed outlier detection mechanism and time-window model which convey the active threat monitor functionality to SYNAPSE.
Datasets D2 and D3 were merged and fed to SYNAPSE. At the end of each day, for all clusters in the current cluster state, we calculated the average WTS and the Jaccard distance between all pairs of clusters. For the latter, we saved the largest value, which corresponds to the most similar cluster pair. Since SYNAPSE's objective is to obtain distinct clusters, each devoted to a single threat, the WTS should always be high (i.e., the elements in each cluster are very similar), and the maximum Jaccard distance should be low (i.e., there are no clusters that should be merged). Figure 4 shows the WTS and maximum Jaccard distance obtained, comparing the proposed DynamicClustream clustering algorithm (DC-WTS and DC-J) to its execution in clustering only mode, without considering re-clustering (NR-WTS and NR-J). The importance of including the re-clustering step (lines 22-25 of Algorithm 1) is clear since it raises the WTS to above 90% independently of the number of clusters and tweets present in the cluster state. The Jaccard distance, although with small values, is higher when using the re-clustering algorithm. Yet, this is an expected result. First, re-clustering produces significantly more clusters, therefore naturally decreasing their degree of separation. Second, since tweets in clusters mentioning different threats are likely to share commonly used security concept words and sentence structure, their similarity is increased.
Regarding the number of clusters obtained using either approach, the re-clustering algorithm naturally increases the number of clusters, as shown in Figure 5. Nevertheless, we argue that in practice, the DynamicClustream algorithm improves the balance between maximising the relevance of the information presented and minimising the time required for its analysis. The WTS results provide guarantees that each cluster has similar tweets, likely about a single threat. Therefore, we can be confident that the set of cluster exemplar tweets provides a complete and accurate summary of the current threat landscape, thus not requiring additional time to analyse more tweets. Without the WTS cohesion validation, each cluster may discuss various threats-a highly plausible assumption based on the very low WTS values in Figure 4 for the NR-WTS case-meaning that all tweets of each cluster would have to be analysed.
End-to-end Benefit. The results presented in Figure 6 highlight the end-to-end benefit of using SYNAPSE, and reinforces the importance of its clustering stage. The figure shows the reduction in the number of tweets that have to be analysed, when compared to the tweet stream, to the classifier output and to the naive Logstash filter described in Section 6. The results show the need for efficient OSINT retrieval tools. Even with the naive keyword-based approach provided by the Logstash filter, the number of tweets marked as relevant would be extremely high, rendering the approach useless to SOC analysts. The introduction of a trained classifier decreases the amount of information by 65%. By attaching a clustering stage, we further reduce the information to be shown by almost 80%, which is a significant improvement.
Active Threat Monitor. To demonstrate the necessity of the active threat monitor implemented by the proposed stream clustering algorithm, we measured the active time for each of the 820 clusters formed during SYNAPSE's operation on the union of datasets D2 and D3. We define the duration of a cluster as the difference in days between the date of its creation and the date of the last added tweet.  would fail to detect active topics through time, since the threat discussion duration varies greatly (between 1 and 57 days), even in a dataset that covers only 100 days.

Analysis of Generated IoCs
Besides the ability to accurately select and aggregate tweets relevant to the security of an IT infrastructure, SYNAPSE provides useful threat intelligence for SOC analysts. To demonstrate this, we present some information about the timeliness, actionability, and relevance of the IoCs generated from the dataset used in previous experiments.
From the data collected over 3 months, SYNAPSE generated 820 clusters (IoCs) containing 1754 tweets. From these, we selected those with 5 or more tweets for analysis, obtaining 65 clusters comprising 466 tweets. These clusters are listed in Appendix B. The remaining 755 clusters have 1 (577 clusters), 2 (101), 3 (55), and 4 tweets (22). Our focus on larger clusters was motivated by the expectation that relevant threats are probably those that attract more attention and, ultimately, are mentioned in more tweets.
All tweets within each cluster were manually analysed. From these, as well as from any hyperlink therein, we extracted all CVEs mentioned (if any) and their Common Vulnerability Scoring System v3.0 (CVSS) [3] impact score, the types of actions that can be performed to respond to the alarm, and a comparison between the date of the earliest tweet in the cluster and the CVE's publication date on NVD. The actionability information was divided into three categories: a patch is available (45 occurrences); a configuration to avoid the vulnerability exploitation is suggested (2 occurrences); and no directly actionable information is provided (14 occurrences). The latter is mostly associated with clusters mentioning exploits to vulnerabilities, with the tweet hyperlinks leading to proofs-of-concept. However, an expert might still make use of this information to prevent exploitation, as discussed in previous work [37]. Patches are mostly announced together with their associated vulnerabilities, regardless of indexing on NVD. In the end, 71% (46) of the clusters provided directly usable intelligence, including exploits whose vulnerabilities were not matched to NVD entries.
Among the 65 clusters, 36 mentioned a total of 122 different CVEs (15 clusters mentioning more than one CVE). Of these, only two have low impact score, about a quarter have medium impact (33), more than half are categorised with high impact (68), and more than a tenth have critical impact (14).
Considering their relevance, 43% (28) of the IoCs were related to CVSS scores above or equal to 7 (high severity) and 12% (8) to scores above or equal to 9 (critical severity). Regarding timeliness, 20% of the alerts were raised 8 days (on average) before their corresponding vulnerabilities were published on NVD.
As an illustration of the richness of the obtained data, Table 6 shows 10 representative IoCs selected from those analysed. In the table, the date column shows the date of the earliest tweet in the cluster and, when a number is shown within parenthesis, it denotes the number of days before publication on NVD. Two additional columns provide information about the threat type (as automatically classified by SYNAPSE) and relevant notes about the cluster content.
From the 10 clusters presented, 6 announce vulnerabilities before publication on NVD, all of them with patches available. Further, 7 are classified with a high CVSS and two with critical impact. For example, the 7 th IoC of the table shows a critical Cisco router vulnerability patched and published three days before its inclusion on NVD. Finally, since not all occurrences are patched at disclosure time, some actionable IoCs contain suggested configurations to avoid exploitations. As an example, the last row in the table shows a WordPress exploit with suggested remediations.
These results show the edge obtained by using Twitter as a security data source. A SOC analyst using SYNAPSE would obtain timely and relevant data about patches to known vulnerabilities, thus possibly reducing the vulnerable system's exposure time. Further, the results also show that vendors publish important impact data before it is included in NVD.

Conclusions
This paper proposes SYNAPSE, a Twitter-based streaming threat monitor for threat detection in security operation centres. It implements a pipeline that gathers tweets from a set of accounts, filters them based on the monitored infrastructure, and classify the remaining tweets as either relevant or not. Relevant tweets are grouped in dynamic clusters and presented as indicators of compromise that can be either manually inspected or fed to SIEMs and other threat intelligence tools. Results show that our system maximises the relevant information (true positive rate of 90%), minimises irrelevant information (false positive rate of 10%), and aggregates related information (only 21% of the relevant tweets are presented). Finally, we performed an evaluation of the IoCs generated by SYNAPSE, showing that highly relevant, timely and actionable information was collected, illustrating the value of our end-to-end approach.

A Pareto figures
Feature extraction. We used Spark's implementation of TF-IDF with default parameters, except for the feature vector size. In order to find a suitable vector size to describe the tweets, eleven values were tested: {30, 50, 80, 100, 200, 300, 500, 750, 1000, 1500, 3000}. This range covers from low to high dimensional vectors, and with it, we should be able to find an appropriate vector size for the datasets.
Classification. As mentioned in Section 3, two classifiers were employed: a linear SVM and an MLP Neural Network. Relevant hyper-parameters and design variables were varied to find a good design for this application. For the SVM, we varied C (the regularization parameter) within {0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1, 2, 5}, and the step size (a parameter for the Stochastic Gradient Descent) within {0.1, 0.5, 1, 1.5, 2, 5}. For the MLP, the number of layers varied from 2 to 8 and the number of neurons per layer within {5, 7, 10, 12, 14, 16, 18, 20}. Each model was evaluated through a 10-fold crossvalidation procedure using dataset D1. The maximum number of training iterations was set to 100 for the SVM and 200 for the MLP, which were deemed to achieve parameter convergence for the range of the design parameters.
To select the best classifiers, we performed a Pareto-optimal search. For each type of classifier we plotted a Pareto front figure (Figure 8 Based on this analysis we select the configurations with the best TPR×TNR balance: those with the smallest distance to the optimum. The best SVM configuration uses a step size and C values of 0.05 and 5, respectively, and the best MLP had 5 layers with 10 neurons each. Both models use feature vectors with a size of 3000, revealing a clear advantage in using high-dimensional feature vectors.
Out of the 13 assets composing the hypothetical IT infrastructure described in Table 2, only 9 (∼ 70%) had related IoCs. The distribution of IoCs over the assets is shown in Figure 9. WordPress is the asset with more related IoCs (19), followed by Linux (14) and Cisco (12). All analysed IoCs mentioned a single asset.
Patch vulnerability A high impact vulnerability (CVSS 7.5) was disclosed and patched before its inclusion on NVD