{'fraction_non_alphanumeric': 0.032816773017319965, 'fraction_numerical': 0.014795596381740411, 'mean_word_length': 5.195482189400521, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CyNER: A Python Library for Cybersecurity Named Entity Recognition

Open Cyber threat intelligence (OpenCTI) information is available in an unstructured format from heterogeneous sources on the Internet. We present CyNER, an open-source python library for cybersecurity named entity recognition (NER). CyNER combines transformer-based models for extracting cybersecurity-related entities, heuristics for extracting different indicators of compromise, and publicly available NER models for generic entity types. We provide models trained on a diverse corpus that users can readily use. Events are described as classes in previous research - MALOnt2.0 (Christian et al., 2021) and MALOnt (Rastogi et al., 2020) and together extract a wide range of malware attack details from a threat intelligence corpus. The user can combine predictions from multiple different approaches to suit their needs. The library is made publicly available.


Introduction
A large amount of open cyber threat and attack information, also called cyber threat intelligence (openCTI), is available on the Internet on platforms such as security blogs, the dark web, software vendors bulletin boards, official news and social networks (Yi et al., 2020). Both structured and unstructured forms of openCTI can benefit security operations center (SOC) analysts and researchers in their increasingly complex jobs of triaging false alerts, finding zero-day attacks, protecting intellectual property, and preventing intrusions from adversaries. However, access to this information from openCTI has remained relatively untapped due to limited research enabling aggregation and structuring of threat intelligence incidents. Furthermore, there is a lack of a gold standard dataset in cybersecurity, which is the first prerequisite for the basic task of information extraction or train any modern natural language processing (NLP) method.
Current methods rely on structured knowledge bases such as the cybersecurity vulnerability enumeration (CVE) 3 database, mitre att&ck framework(att&ck) 4 to extract named entities which result in high accuracy but produce poor performance when similar entities are extracted from unstructured, complex threat reports, which is more often the case. Conditional random fields (CRF) (Joshi et al., 2013), support vector machine (SVM) (Deliu et al., 2017), and many learning models have been unsuccessful in yielding satisfactory results in extracting cybersecurity events from openCTI. While transformer-based methods, such as BERT (Devlin et al., 2018), have been explored, the cybersecurity concepts extracted provided a limited advantage over information extracted from structured sources such as CVE (Xie et al., 2021).
To address the challenges above, we propose CyNER, a generic python library for extracting cybersecurity threat intelligence incidents. Specifically, we make the following contributions: 1. We provide a manually-labeled, benchmark dataset annotated on a wide range of cybersecurity incidents. While the annotated corpus relates to Android malware threat analysis, the concepts are generic enough to apply to other security threats. We also provide a corpus of noise-free, high-quality openCTI reports written on android malware analysis.
2. We provide CyNER, an easy to use python library that allows access to transformer-based NER models pre-trained on high-fidelity security events (e.g., malware, vulnerability) and a heuristic model for threat indicators (e.g., malware hashes, compromised IP addresses).
In addition, we include support for extracting generic entity types with popular NLP libraries.
3. We provide a flexible and modular implementation of CyNER and user documentation.
The library provides the option to combine prediction from multiple models with configurable priority settings.
Our proposed framework, CyNER allows cybersecurity information extraction using NER models that deliver high accuracy. The malwarerelated threat intelligence discovery combines malware data with general world concepts, integrates "knowledge," correlation, and concept grouping, and therefore, generates inference and insights for SOC analysts.
In this work, we instantiate existing malware ontologies (Rastogi et al., 2020;Christian et al., 2021) using a corpus comprising over 60 high quality and large threat intelligence reports written between 2018-2021 and downloaded from websites belonging to security and technology organizations (Kaspersky, Symantec, McAfee). The annotated text contains entities manually generated by graduate students trained by their advisor experienced in cybersecurity. We build on the existing strengths of the security, semantic web, and natural language domains. Security ontologies (Rastogi et al., 2020) (Swimmer, 2008) are extended to extract cyber security threat intelligence from online sources. Other benefits include the interoperability of information across applications and creating labeled features for machine learning models. Using ontologies when labeled data allows queries over large datasets to identify trends and predict future occurrences.

Related Work
Security ontologies (like malware) have attempted to provide a structure to security concepts that capture event analysis (Swimmer, 2008) primarily to collate data and distribute it across organizations and institutions (Barnum, 2012;Connolly et al., 2014). Other models for extracting and structuring threat intelligence focus on one of the followingidentifying attack patterns, software vulnerabilities, threats, or information dissemination. CVE 5 , NVD 6 are vulnerability tracking programs where the security community feeds information. For sharing threat intelligence, there are industry standards like Structured Threat Information eXpression (STIX) (Barnum, 2012), which provides a language-agnostic framework to capture threat intelligence into a shareable package. In contrast, a trusted automated exchange of indicator information (TAXII) (Connolly et al., 2014) is a platform that can send and receive the STIX package.

System Overview
The primary goal of our library is to provide cybersecurity researchers with models for extracting cybersecurity entities and provide the framework to train new models on annotated datasets. We show a detailed system diagram of our library in Figure  1. In this section, we discuss the three key components of our framework -using pretrained models for prediction, training models on new datasets, and augmenting prediction using a combination of different models.

Pretrained Model for Prediction
We provide three different types of models that users can readily use to extract cybersecurity entities-

Transformer models pretrained on cybersecurity corpus
We train transformer models on an annotated cybersecurity corpus and add the pretrained models in our library. Our dataset consists of five classes-malware, indicator, system, organization, and vulnerability. These models allow users to extract named entities related to the cybersecurity domain. We use the tner library (Ushio and Camacho-Collados, 2021) to train the models.

Heuristic for indicators of compromise
We provide a heuristic-based approach for categorizing different indicators of compromise, e.g., URL, IP address, hash, etc. These entities do not require an understanding of the context they appear in and can be usually extracted with regular expression matching. We use regular expression templates proposed in earlier work on cybersecurity NER (Yi et al., 2020;Piplai et al., 2020). We show some example regex patterns in Table 1.

Generic NER models
Sometimes entities that do not fall under cybersecurity concepts may be of interest when analyzing threat reports. For example, extracting targeted countries of a malware attack from threat reports can give important insight regarding its behavior.
Off-the-shelf NER models provided in popular NLP libraries, which are trained on generic corpus like Ontonotes (Hovy et al., 2006) can achieve good accuracy on such classes of entities (Kim et al., 2020). We integrate Flair (Akbik et al., 2019) and spaCy 7 with our library, and users can readily use the pretrained NER models available in those libraries.

Model Training
We provide modules for finetuning transformer language models on user-provided datasets. We use Huggingface's transformer library (Wolf et al., 2019) for finetuning. The user can use pretrained language models available in the transformer library and set different hyperparameters during model finetuning.

Combining Prediction from Multiple Approaches
We provide a simple yet intuitive mechanism for combining predictions from different methods. The user can use any combination of the available models (transformers, heuristics, flair, spacy) for making a prediction. The user can also define priority when merging outputs from multiple models using  a keyword argument priority. The default value of the priority is 'HTFS,' which indicates that the output from the heuristic approach will be given the highest priority, followed by the prediction from the given transformer model, followed by the Flair and Spacy models (if provided). If there is an overlap between predicted entity positions from multiple approaches, we use the prediction from the highest priority one.

Datasets
We collected approximately 60 threat intelligence reports referenced in the MITRE att&ck website under the software category. 8 Each threat report describes a unique malware. Since cleaning unstructured CTI reports can be very challenging (Kim et al., 2020), we manually extract the clean text from each report. We use the BRAT annotation tool (Stenetorp et al., 2012) for annotating the corpus. Example annotation is shown in Figure 2. Our dataset consists of five classes of high importance for analyzing threat reports -malware, indica-   tors of compromise, system, organization, and vulnerability. The malware class encapsulates viruses, trojans, ransomware, etc. Operating system (e.g., Android, Windows), software (e.g., Adobe flash player, Skype), and hardware. Indicators of compromise comprise domain name, URL, IP address, filename, Hash, email, port number, etc. Vulnerability includes both CVE ID (e.g., CVE-2012-2825) and mention of exploits (e.g., master key vulnerability). It is worth noting that other types of entities like location and person are also important when analyzing threat reports. However, these and other generic entities can be extracted well using different off-the-shelf models (Kim et al., 2020). As a result, we added the support to extract such entities using other NLP libraries and did not include them in our annotation. We split the corpus into train, dev, and test split containing 40, 10, and 10 documents, respectively. Statistics of different types of entities in each split are displayed in Table 2. We convert the annotation in BIO format for training, B-prefix indicating the beginning of a tag and the I-prefix indicating the inside of that tag. An outside (O) tag is used to indicate a token that does not belong to any of the five categories. We have 106991 total tokens in the complete corpus, with 4530 tagged entities.

Experimental Settings
We finetune transformer models available in Huggingface's transformers library (Wolf et al., 2019) for sequence tagging. We train both base and large variants of BERT (Devlin et al., 2018), RoBERTa   (Liu et al., 2019) and XLM-RoBERTa (Conneau et al., 2019) models. We add a linear layer to the hidden layer representation obtained from the transformer model for the token classification task. We train each model for 20 epochs with a sequence length of 128. We use an initial learning rate of 1e-5 for the base models and 5e-6 for the large models. We use 32 samples for each mini-batch. All models are trained using AdamW optimizer (Loshchilov and Hutter, 2017) on a single Nvidia Tesla V100 GPU.

Results
We report the span micro-F1 score computed by the seqeval 9 library. We report results for the six different transformer models on Table 3. XLM models outperform both BERT and RoBERTa models, despite being multilingual. XLM-RoBERTalarge model achieves the best average F1 score of 76.66%. Class-specific precision, recall, and F1-score are displayed in Table 4. The model performs relatively poorly on the Organization class compared to others.
6 System Demonstration

Installation
CyNER is available as a python framework in Github. Users can run the following command for installation:  In listing 2, we combine prediction from the same transformer model with the heuristic-based approach. We use priority 'HTFS,' which means prediction from the heuristic-based detection will be prioritized. The only difference from the previous output is that the hash is now detected as SHA256, a subcategory of the Indicator obtained from regular expression matching. Since the output from the transformer overlaps with the detection from the heuristic and has a lower priority, it is omitted from the merged prediction.
In listing 3, we add the NER model from Flair on top of transformer and heuristic approaches. 10 Note that we do not use the Spacy model by default, so it is not used for this prediction. Using the Flair NER model allows us to capture two more generic entities -'German-language' as MISC (Miscellaneous) and 'UK' as LOC (Location).

Fine tuning Models
Users can fine-tune transformer language models on annotated NER datasets. The dataset needs to be annotated in the CoNLL 2003 as shown below:

Conclusion
In this paper, we have presented a python library for extracting cybersecurity named entities. We have provided pretrained transformer-based NER models trained on annotated threat reports. In addition, we have added a heuristic-based approach for extracting different indicators and incorporated popular NLP libraries that will allow the user to extract more generic entity types. We have added flexible options to enable the user to combine predictions from multiple models with a defined priority order and support finetune language models on annotated NER corpus. We believe this library will benefit cybersecurity researchers and practitioners to analyze cybersecurity threat reports and gather meaningful insights.