{'fraction_non_alphanumeric': 0.03620071684587814, 'fraction_numerical': 0.024253285543608123, 'mean_word_length': 4.595588235294118, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Searching for Multi-Fault Programs in Defects4J

Defects4J has enabled numerous software testing and debugging research work since its introduction. A large part of its contribution, and the resulting popularity, lies in the clear separation and distillation of the root cause of each individual test failure based on careful manual analysis, which in turn allowed researchers to easily study individual faults in isolation. However, in a realistic debugging scenario, multiple faults can coexist and affect test results collectively. Study of automated debugging techniques for these situations, such as failure clustering or fault localisation for multiple faults, would significantly benefit from a reliable benchmark of multiple, coexisting faults. We search for versions of Defects4J subjects that contain multiple faults, by iteratively transplanting fault-revealing test cases across Defects4J versions. Out of 326 studied versions of Defects4J subjects, we report that over 95% (311 versions) actually contain from two to 24 faults. We hope that the extended, multi-fault Defects4J can provide a platform for future research of testing and debugging techniques for multi-fault programs.


Introduction
Defects4J [9] is one of the most popular real-world Java fault datasets in the field of software engineering, with over 650 citations as of June 2021 since its publication in 2014. Defects4J provides a number of software faults, along with a clearly separated and isolated set of test cases that can reveal each fault, making it easier for researchers to study individual faults in isolation. Due to both the ease of use and the realism of the curated faults, it has been broadly adopted in the empirical validation of numerous automated debugging work such as Fault Localisation (FL) [2,13,17] and Automated Program Repair (APR) [4,11,15].
However, in realistic debugging scenarios, multiple faults can coexist in software and affect the test results together. For example, a Continuous Integration (CI) process of large-scale industry software can produce hundreds of failing test cases that are caused by distinct root causes [7]. The isolation of individual faults that made Defects4J compatible with the Single Fault Assumption (SFA) ironically prevents it from being used to study the debugging of multiple faults. According to a systematic literature review of multiple faults localisation [19], the majority (33) of the 55 selected studies used only C faults for the evaluation. Only ten studies are reported to consider Java programs, five out of which employ Defects4J [12,14,18,20,21]. Only Zheng et al. [21] combined separate multiple Defects4J faults; since the procedure of creating the multiple faults was manual, only 46 have been created. The remaining work either concern multi-hunk faults, i.e., a single fault that can only be fixed by changing multiple locations [16] and consequently use Defects4J as it is [18,20], or actually concern neither multiple faults nor multi-hunk faults [12,14]. Note that, in this paper, we use the term multiple faults to denote the faults that can be fixed independently of each other.
Given the contributions to the automated debugging research made by De-fects4J under SFA, we believe that the study of automated multi-fault debugging techniques [19], such as failure clustering [5,7,8] or fault localisation for multiple faults [1,6,21], would significantly benefit from the construction of a reliable dataset of realistic multi-fault Java programs. In this paper, we build a realworld Java multi-fault dataset by extending Defects4J. Instead of artificially injecting mutation or manually grafting faults, we use iterative search to systematically detect the existence of multiple faults in each version via fully automated transplantation and execution of the fault-revealing test cases. We report that 311 out of 326 studied faulty versions (95.4%) contain multiple faults, ranging from two to 24. The result data and replication package are publicly available 1 .

Proposed Approach
The faults in Defects4J are extracted from the actual development history of various projects. Since every fault has a different life span [3,10], even a fault that was recently fixed may have existed in the project for a long time. In this work, we check if a specific fault N in version P of a Defects4J subject exists in an older version P containing another fault M . If N exists in P , we regard P as a multi fault program that includes both N and M . Note that we modify neither P nor P : the check is performed by test transplantation, and therefore we only reveal what already exists in P . The following sections present the motivating example and our proposed method to search for multi-fault programs.

A Motivating Example
Listing 1.1 shows the fault Math-5 in Defects4J and its developer patch changing the return value from NaN to INF. 2 This fault is revealed by the test case testReciprocalZero (Listing 1.2) that checks if the return value is equal to INF. Each Defects4J fault is similarly provided with a set of fault-revealing test cases that reveals a single fault.
We note that, with few exceptions of recently added subjects and versions, the majority of faulty versions in Defects4J are indexed chronologically based on 1 ---a / src / main / java / org / apache / commons / math3 / complex / Complex . java 2 +++ b / src / main / java / org / apache / commons / math3 / complex / Complex . java 3 @@ -304 ,7 +304 ,7 @@ @@ public Complex reciprocal () { their revision dates, so that a lower ID refers to a more recently fixed fault: for instance, Math-5 was fixed later than Math-6. Therefore, the faulty source code version of Math-6 (referred to as Math-6b) may also contain the fault Math-5. Listing 1.3 confirms that Math-5 does exist in Math-6b, but is simply not revealed due to the absence of the fault-revealing test case, testReciprocalZero. When transplanted to Math-6b, the test fails with the same error message as in Math-5b, showing that Math-6b contains at least two faults, Math-5 and Math-6.

Implementation Details
The process in Section 2.2 is dockerised and automated. We use javaparser 4 to detect the line range of the target test methods during transplantation. In the docker container, one can simply checkout to the multi-fault version by invoking python3.6 checkout.py Math-1-2-3 -w /tmp/Math-1-2-3, after which the same source code with Math-3b, augmented with the fault-revealing test cases of Math-1 and Math-2, is checked out.

Results
Multiple Fault Subjects Figure 1a shows how many faults are contained in the faulty versions of five projects 5 . The x-axis shows the number of faults found in each faulty version, and the y-axis shows the number of faulty versions. Out of 326 faulty programs, 95.4% (=311/326) of them contain multiple faults (i.e., # found faults > 1). Furthermore, 126 and 22 faulty versions have ≥ 10 and ≥ 20 faults, respectively. For example, Closure-90b contains 24 faults. Our repository contains the full results of the found multi-fault versions.
Lifespan of Faults To confirm whether lifespans of Defects4J faults vary similary to existing findings [3,10], we calculate the lifespan of each fault. Let us define the lifespan of fault N as the number of days between the date of the oldest previous faulty version where fault N is detected and the revision date of N when the patch is applied. If there is no preceding version where the fault N is revealed, the lifespan is zero. Figure 1b shows that lifespans of faults range from 0 days up to longer than three years (e.g., Lang-41 has the lifespan of 1,187 days). The variance in lifespan suggests that the probability of having multiple faults at any given time can be nontrivial.

Conclusion
The paper presents a multi-fault Java dataset based on Defects4J, for which subjects with multiple real faults are constructed by transplanting tests without modifying the source code. Exploiting the chronological indexing of Defects4J, we propose a systematic search strategy to find co-existing faults that have not yet been revealed by failing tests. The results show that 311 out of 326 versions in Defects4J actually contain multiple faults. We hope that our extension of Defects4J can aid future research on search-based automated debugging under the existence of multiple faults.