{'fraction_non_alphanumeric': 0.0354335194257719, 'fraction_numerical': 0.009722380464091756, 'mean_word_length': 4.977752553916004, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
ROMEO: Exploring Juliet through the Lens of Assembly Language

Automatic vulnerability detection on C/C++ source code has benefitted from the introduction of machine learning to the field, with many recent publications considering this combination. In contrast, assembly language or machine code artifacts receive little attention, although there are compelling reasons to study them. They are more representative of what is executed, more easily incorporated in dynamic analysis and in the case of closed-source code, there is no alternative. We propose ROMEO, a publicly available, reproducible and reusable binary vulnerability detection benchmark dataset derived from the Juliet test suite. Alongside, we introduce a simple text-based assembly language representation that includes context for function-spanning vulnerability detection and semantics to detect high-level vulnerabilities. Finally, we show that this representation, combined with an off-the-shelf classifier, compares favorably to state-of-the-art methods, including those operating on the full C/C++ code.


I. INTRODUCTION
Machine learning has advanced automatic vulnerability detection significantly compared to traditional static analysis security testing (SAST) tools [1]. The focus of recent methods is mainly on C/C++ source code, where many vulnerabilities can be detected reliably, but others cannot [2]. We argue that the C/C++ compiler's output warrants the same attention as its input, if not more. The machine code is more closely related to the actual state transitions during execution than the original source code, and SAST tools often fail to consider implementation details of compilers [3]. Moreover, source code is not always available for third party software, which nevertheless has to be audited. Finally, a method that works directly on assembly language or machine code can easily be transferred to a dynamic environment, e.g. for runtime monitoring of JIT compilers, and it supports all compiled higherlevel languages by design. While there is recent work in the area of vulnerability detection on machine code and assembly language (see section II-A), there is no reproducible, publicly available dataset for this task. To the best of our knowledge, there is also no binary vulnerability method that incorporates the semantics and context information necessary to detect more abstract vulnerabilities spanning multiple functions.
To advance the state of the art in this direction, we contribute the following: • ROMEO, a binary vulnerability detection benchmark dataset based on the Juliet test suite [4] [5]) categories. We publish the source code of the entire processing pipeline, which is reproducible and configurable to meet the needs of other researchers.
• A simple text representation of disassembled binaries suitable for various sequence classifiers. It incorporates context for across-function vulnerabilities and preserves semantics to identify API calls, while still preventing label leakage.
• Experiments using an off-the-shelf Transformer approach [6]. We show the efficacy of our assembly language representation, which compares favorably even to stateof-the-art methods that have access to the full source code, and highlight the relative strengths and weaknesses in qualitative analyses.

II. RELATED WORK
In this section, we review previous work related to our investigation in two main areas. We consider approaches to vulnerability detection that use an assembly language or machine code representation, and we explore work that uses the Juliet test suite as a benchmark or evaluation dataset.

A. Assembly Language in Vulnerability Detection
Lee et al. [7] combine a bespoke encoding of assembly language instructions called Instruction2vec [8] with a deep learning model "Text-CNN" to detect vulnerabilities in binary code. The encoding represents each instruction as a vector of a fixed length. Individual components of the instruction, i.e. the opcode and operand fragments, are encoded using a custom word2vec [9] model. BVDetector [10] operates based on pre-extracted program slices. It relies on a per-token word2vec encoding. While the authors test a various neural networks to classify the encoded slices, they find that a BGRU [11] performs best. They measure the performance of their method on program slices extracted from a subset of Juliet concerning memory corruption and number handling. We construct a subset using the same criteria to compare our approach to BVDetector as their dataset is not publicly available. The notion of program slices [1], [12] is extended to assembly language by Li et al. [13], who incorporate a combined representation of source code and assembly called "hybrid slices". Le et al. propose a Maximal Divergence Sequential Auto-Encoder [14] and rely on a fixed encoding of opcodes and a histogram-like encoding of operands to represent assembly language instructions.
In [15], [16], a dataset called BVATT is constructed with similar goals to ours. It is intended as a reliable benchmark for binary vulnerability detection methods, but it is no longer publicly available.

B. Juliet Test Suite as a Benchmark for Vulnerability Detection
Russel et al. [17] [12].
BVDetector [10] uses a subset of the Juliet test suite without any additions (see section II-A for further details). A similar approach, but combined with the C/C++ source code, is taken by Li et al. [13]. The Juliet test suite is further used as an alternative training and evaluation dataset in the REVEAL study [2] by Chakraborty et al., who criticize the high number of duplicates compared to real-world datasets. We describe our approach to this matter in section III-D1.

III. CONSTRUCTING ROMEO FROM JULIET
This section details the steps required to construct our assembly language dataset ROMEO from the Juliet test suite, guided by the following design goals: a) Coverage: The dataset should cover as large a fraction of the Juliet test suite as possible. While previous work [7], [10] focuses on subsets of the test suite to manage complexity, or uses static analysis rules to extract examples [1], this work covers the largest possible fraction of the entire test suite that does not compromise the following two properties and is compatible with Linux environments. b) Context: While our goal is to provide examples on a function level, a single function can be vulnerable or not depending on the content of related functions or data. The dataset should provide appropriate context in terms of relevant functions for each example. c) No label leakage: Because of the very methodical construction of the test cases in Juliet, there are various ways in which label information can leak into examples. Symbols include label information in their names, e.g., when they end in good or bad. Context can also leak information, e.g., the presence of any context is enough to distinguish a primary good function from a primary bad function in certain flow variants. No label leakage should occur.

A. Preparations
Before we can disassemble examples, the Juliet test suite requires preparations, which we discuss in the following. 1) Modifications: As a sanity check, the support file io.c, which we need to include for context, contains 18 empty functions by the names of good1-good9 and bad1-bad9, which we remove. Otherwise, they would be present in every testcase after linking, and could not be distinguished from testcase functions that are empty on purpose, concerning CWE-570 (Expression Always False). Hence, this modification is unavoidable. Aside from these support functions, we do not modify the Juliet source code in any way.
2) Compilation and linkage: We compile all testcases individually with GCC 11.2.0 using the provided Makefile generation script. Since our platform of choice is Linux, the script does not create Makefiles for Windows-specific CWEs such as CWE-247 (Reliance on DNS Lookups in a Security Decision). We link each testcase into a single object file to capture all relevant functions. Overall, this process results in object files for 41,812 testcases covering 91 CWEs, with the remainder consisting of Windows-specific testcases.

B. From C/C++ to Assembly Language
We compile the sources fully into machine code, link them, and then disassemble the resulting binaries. This can in principle reveal more information about optimizations, e.g., linktime optimization (LTO), as well as specific instruction encodings, than compiling to assembly directly. It also more closely resembles a black-box analysis situation. We disassemble each testcase's linked object file using objdump configured to produce Intel syntax assembly, and obtain demangled symbol tables. The results are then verified against capstone.

C. Extracting Examples
In the following, we construct the ROMEO assembly language representation from the disassembly in a number of steps, including modifying operands and symbols, and selecting relevant context. Furthermore, we describe the mapping from testcases in Juliet to examples in the ROMEO dataset.
1) Symbol representation: We build a scrambling table for each testcase, mapping each local symbol to a unique random name in the pattern of lc000-lc999 to prevent label leakage, e.g., from function names ending in good or bad. Global symbols are not renamed, as they contain no label information, but are crucial for vulnerability detection, e.g., memcpy.
Many instructions contain addresses in the operands, which objdump annotates with corresponding symbols. As the specific memory layout is not important and potentially confusing for a classifier, we remove the addresses. Memory operands are represented only by symbols, which we replace with their scrambled names.
Moreover, many operands represent memory locations indirectly, e.g., as offsets to registers. When objdump recognizes a known address, it emits a comment containing the address and a symbol plus offset. We replace the corresponding operand with the symbol and offset, also scrambled if applicable. The lea instruction in listing 1.2 illustrates this replacement.
2) Selecting and representing Functions: Every function in the .text section of an object file is a potential example for ROMEO, or part of the context of one (see section III-C3). Hence, we extract a text representation of each function, consisting of a header line with the function's (scrambled) name, followed by the disassembly modified in the aforementioned manner. The function name is prefixed with an exclamation mark, such that there is a unique token to mark the beginning of a function. Listing 1 shows an example of this representation.
However, not all functions are either positive or negative examples of a vulnerability. There are also supporting or completely unrelated functions. We only admit examples into the dataset that are primary or secondary good functions, or primary bad functions according to the regular expressions in [4]. We further remove the primary good function to avoid label leakage (see the introduction to section III).
3) Including context: The Juliet test suite lends itself to vulnerability classification on a function level, which we adapt for ROMEO. However, individual functions cannot always be classified based on their body alone. In terms of the source and sink model, a function containing a bad source might call another function containing the corresponding bad sink. Its vulnerability status can then only be determined by analyzing the called function as well.
To mitigate this issue, we include context information by concatenating the text representation of a given function with the text representation of all functions that are referenced by it, recursively (see listing 1). We exclude boilerplate and runtime functions, and remove all bad functions if the given function is a primary or secondary good function, and vice versa, to avoid label leakage.

D. Building a Dataset for Machine Learning
The collection of examples from the previous section is then labeled either with binary ("good" or "bad") or multi-class (CWE number or "no weakness") labels as chosen by the user. For our evaluation, we use the binary labels because most related work uses this formulation. Subsequently, we eliminate duplicates and split the dataset for proper validation.
1) Duplicate Elimination: Duplicates in benchmarks based on the Juliet test suite are observed by several works [18], [2], [17]. Since each testcase in Juliet is initially roughly unique, duplicates likely result from preprocessing or extraction steps, or in our case, from the compilation process. Our proposed ROMEO representation leads to a fraction of 2.6% (with context) and 9.7% (without context) duplicate examples, i.e., examples with an identical representation.
For each set of identical examples, we remove all but one instance. Random selection determines which exact instance of a duplicate example is kept.
Our fraction of duplicates is lower compared to other works, e.g., [17], which identifies an extreme value of 90.2% for Juliet. However, they do not remove the primary good functions of each testcase beforehand, which are mostly identical. Moreover, their representation is designed to require a very small vocabulary, and removes most natural language elements such as identifiers and literals. And most importantly, it does not consider context, which would help distinguish between similar instances. An overview of duplicate fractions in vulnerability detection benchmarks can be found in [2].
2) Splitting: We then randomly split the dataset into three parts for training, validation and testing, with a fraction of 80%, 10% and 10% of the examples, respectively. The resulting dataset is prepared for use by any machine learning methods capable of processing text sequences. We name it ROMEO to highlight its dependence on Juliet and make it available publicly 1 .

IV. EXPERIMENTS
We pose research questions relating to the design goals of ROMEO and the usefulness of its representation for vulnerability detection applications as follows: RQ1 What benefits and drawbacks are associated with the inclusion of context (as described in section III-C3)? RQ2 To what extent, if any, does our assembly language representation leak label information? RQ3 How does an off-the-shelf model using our assembly language representation compare to other methods, including ones with access to the C/C++ source code? In the remainder of this section, we first describe our experimental setup including datasets and baselines. We then Listing 1: A function extracted from a Juliet testcase concerning CWE-191 (Integer Underflow) and its accompanying context. Both are in the text representation as described in section III-C. This example illustrates one purpose of including context functions, namely to check whether a result is actually used, e.g., in an API call. present our results structured along the aforementioned research questions and offer brief discussions. Finally, we summarize the results, linking questions and answers, and address the limitations of this evaluation.

A. Setup
This section describes the situation concerning binary vulnerability detection datasets derived from Juliet as well as our Transformer-based vulnerability detector.

1) Datasets:
In section II, we list several works that perform vulnerability detection on assembly or machine code representations of the Juliet test suite. In particular, there is [14], which is only available in an already encoded vector form, from which the original instructions cannot be recreated. The same is true for [7]. BVATT [15], [16], is not available publicly anymore, against the claims in their work.
In all cases, the respective authors did not accommodate our request for the datasets in their original form.
2) Baselines: While we cannot directly compare our representation combined with the Transformer-based approach (see section IV-A3) to other methods on identical data (due to availability reasons explained in section IV-A1), we can draw conclusions from comparisons to other work on reasonably similar data. Specifically, we compare our approach to works that evaluate their methods on test sets derived from the Juliet test suite, namely Instruction2Vec [7], [8], BVDetector [10], Russel et al. [17], and REVEAL [2]. We modify the ROMEO dataset to match the respective subset or construction as closely as possible in each case.
3) Transformer-based model for vulnerability detection: The empirical part of this work focuses mainly on exploring the advantages and drawbacks of an assembly language representation for vulnerability detection. Hence, we select a rather generic, but very powerful approach to perform the actual classification task, namely Transformers [19]. Specifically, we use the pre-trained CodeBERT model [6] for initialization. Our implementation is based on PyTorch [20] and HuggingFace Transformers [21]. We use a byte-pair encoding [22] optimized on the ROMEO training set, which can represent common  We train the model for ten epochs on the ROMEO training set with and without context, respectively. One epoch equals one iteration over the entire training set. We use a minibatch size of 16, a learning rate of 1.1e-5, and an L2 regularization coefficient of 3e-4. These values are determined using the validation set, and subsequent evaluations are performed on the held-out test set. Each training and evaluation is performed three times, and all reported results are the average over all runs, with standard deviations displayed where applicable.

B. Results
In the following, we present our experimental results to answer our research questions. All metrics are reported in terms of a binary classification problem (vulnerable / not vulnerable) for comparison with competing methods. 1) Context: RQ1 asks "what benefits and drawbacks are associated with the inclusion of context?" We answer this research question quantitatively and qualitatively using two variants of the ROMEO dataset, one with context and one without. Context is defined in section III-C3.
We first apply our ROMEO method (see section IV-A3) to both variants of the ROMEO dataset. With context, the overall accuracy on the held-out test set is 96.9% and the overall F1 score is 94.0%. Without context, the accuracy and F1 score are 90.2% and 81.9%, respectively. Hence, on average, our method strongly benefits from the included context.
From a machine learning point of view, removing the context information can be interpreted as manual feature selection, which is occasionally done on purpose to prevent overfitting. Removing features that are unrelated to the problem prevents the classifier from adapting to spurious correlations. However, in our case, there are clearly instances where context is crucial to determine whether a function is vulnerable or not, e.g., when the actual vulnerability is spread over multiple function calls. Still, context information might not always be relevant and introduce misleading features. To get a clearer picture of this situation, we evaluate our method for each flow variant and CWE individually. Table II shows the accuracy obtained with and without context information and which flow variants are most affected. In line with our expectations, the flow variants that benefit most from including context all describe vulnerabilities spread over multiple functions or even multiple translation units. Here, we see accuracy improvements up to 34.5 percent points. Without context, it is not possible to identify whether a function is calling a "bad sink" (assuming no label leakage through symbols or inlining by the compiler). We also show the most negatively affected flow variants, and in the worst case, the difference in accuracy is less than two percent points.
In table I, we present the CWE types most affected by the inclusion of context. Overall, the most positively affected CWE types are simply those with the most examples and vice versa for the most negatively affected types. The most negatively affected weakness types have such low representation in Juliet that we cannot extract a meaningful interpretation of the results. In contrast to the flow variants, there is no set of CWE types that specifically benefit from context inclusionmost of them do.
To answer RQ1, including context in the assembly language representation of ROMEO is clearly beneficial for vulnerability detection. The benefits and drawbacks of including context do not appear to be specific to certain types of weaknesses in terms of CWEs. They are specific to certain presentations of vulnerabilities, i.e., flow variants, that are spread out over multiple functions or translation units.
2) Label leakage: RQ2 asks "To what extent, if any, does our assembly language representation leak label information?" Preventing label leakage is also one of the design goals of ROMEO (see section III).
Our process is carefully designed to prevent label leakage in terms of features, e.g. by renaming symbols (section III-C1) and excluding telltale functions from the context (section III-C3). Still, we should verify that there is no  3) Comparison with other methods: RQ3 asks "How does an off-the-shelf model using our assembly language representation compare to other methods, including ones with access to the C/C++ source code?" As alluded to in section IV-A2, we offer an estimate of relative performance. All performance numbers that we use from other works are based on the same Juliet test suite, but there are different evaluation processes. The works differ in granularity, i.e., program slices [10] vs. functions [7], [17], [2]. Moreover, we extract functions from the full dataset unconditionally, but popular options include a selection of candidates using SAST. Deduplication occurs at different stages or according to different representations [17], [2], or not at all [7], [10]. With these limitations in mind, we offer comparisons to related methods [7], [17], [10], [2] and modify our dataset to match their evaluations as closely as possible.
The combination of Instruction2Vec and Text-CNN [7], [8] is evaluated on a small subset of the Juliet test suite, namely vulnerabilities of type CWE-121 (stack-based buffer overflows). It also relies on assembly language instructions instead of source code. For comparison, we evaluate our Transformerbased method on this single vulnerability category only. Our ROMEO method achieves an accuracy of 97.3%, compared to 96.1% reported by Lee et al. Both methods operate on a function level.
Russel et al. [17] employs a bespoke C/C++ lexer to obtain a token sequence directly from the source code. On the Juliet suite, which is not subsampled in their work, they report an F1 score of 84.0% for the best combination of classifiers. In contrast, the ROMEO method obtains an F1 score of 94.0% over the whole suite (see also table III).
BVDetector [10] extracts program slices instead of functions, but operates on assembly language instructions. The authors evaluate their method on a subset of Juliet involving memory corruption (MC) and number handling (NH) vulnerability types as defined by the STONESOUP program 2 . We evaluate our ROMEO method on a subset corresponding to these types and obtain an accuracy of 97.1% on MC and NH combined, where BVDetector reaches 96.7%. Both subsets are evaluated individually in table IV. REVEAL [2] is proposed alongside an in-depth analysis of the current state of deep learning-based vulnerability detection and an eponymous dataset. It provides interesting insight into the limitations of synthetic datasets such as Juliet, but nevertheless the authors evaluate their method on it. On the unmodified dataset, they report an F1 score of 93.7%, compared to ROMEO's very close 94.0% (see also table III). However, due to the extreme class imbalance observed by them, they apply SMOTE to alter the distribution by resampling that data and increase their F1 score to 95.7%.
Overall, the ROMEO method, consisting of our assembly language representation combined with an off-the-shelf Transformer approach, performs better than other state-of-the-art methods, except for REVEAL-SMOTE, to the degree that these methods can be compared fairly. Importantly, the outperformed methods include [17], [2], which have access to the full C/C++ source code including comments.

C. Limitations of this Study
There are aspects in which this investigation is limited. For example, we evaluate on the function level, similar to [7], [17], [2]. This design choice is appropriate for Juliet as well as ROMEO, but may not translate well to real-world applications.
We maintain that Juliet is a sensible choice for our investigation as it allows for a methodical and detailed evaluation of our representation. It covers a considerable range of weaknesses. However, it is not representative of real-world software in terms of vulnerability statistics and code complexity.
Moreover, in its present state, the examples in ROMEO only include information from .text sections. Hence, information from data segments is missing. In listing 1, this can be seen in the context function lc18. The format string handed to printf is not part of the representation.
Currently, comments from the C/C++ code are also not carried over to the assembly language representation. While this could be solved by accessing debug information, it is not reasonable to expect this information to be present in a realworld binary analysis scenario. Our choice of x86-64 ISA is another minor limitation, however, our representation is not technically restricted to it and could be adapted, e.g., to ARM. 2 https://samate.nist.gov/SARD/around.php V. CONCLUSION In this work, we present ROMEO, a publicly available, reproducible and reusable binary vulnerability detection benchmark dataset. ROMEO is derived from the Juliet test suite [4] of C/C++ vulnerabilities by compiling and extracting an assembly language representation. We combine our representation with an off-the-shelf Transformer model, and compare the combination with other methods. Our experiments show that even methods with full access to the C/C++ code rarely outperform it, highlighting the relative importance of observing compiler and linker closely.