{'fraction_non_alphanumeric': 0.042630503195665574, 'fraction_numerical': 0.023073678746390682, 'mean_word_length': 4.5471331389698735, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A systematic mapping study on cross-project defect prediction

Cross-Project-Defect Prediction as a sub-topic of defect prediction in general has become a popular topic in research. In this article, we present a systematic mapping study with the focus on CPDP, for which we found 50 publications. We summarize the approaches presented by each publication and discuss the case study setups and results. We discovered a great amount of heterogeneity in the way case studies are conducted, because of differences in the data sets, classifiers, performance metrics, and baseline comparisons used. Due to this, we could not compare the results of our review on a qualitative basis, i.e., determine which approaches perform best for CPDP.


Introduction
In order to utilize the often limited resources available for the quality assurance of a software efficiently, test managers require tools that support decision making regarding the focus of the quality assurance. One such tool is defect prediction, i.e., the prediction of the location of defects 1 . In recent years, the prediction of defects in a target product based on data from other products, the so called Cross-Project Defect Prediction (CPDP) became a popular research topic. Within this article, we provide a systematic mapping study of the literature onon CPDP. Three previous literature reviews were performed in the area of defect prediction. Our mapping study differs from the others in the following aspects: -Timeframe: the previous reviews cover the time up to 1999 (Fenton and Neil, 1999), for the timeframe 2000-2007 , and the timeframe 2000-2010 . Our study covers the timeframe 2006-2015, whereas 2006 is the year of the first publication we identified that specifically addressed CPDP. -Focus: The previous reviews were focused on defect prediction in general.
Moreover, the topic CPDP was very new in the previously considered timeframes. The focus of our review is strictly on CPDP, which was only a sidenote or too new to be covered by the previous reviews. -Analysis: Following the guidelines by Kitchenham and Charters (2007) we performed a detailed analysis of the state-of-the-art of CPDP to provide an in-depth look into the approaches proposed, state of practice of case studies, and the results achieved.
We make five significant contributions by presenting: -A set of 50 studies addressing CPDP published from July 2002 to December 2015.
-A summary of the approaches proposed for CPDP which can be used by researchers as foundation for future investigations of CPDP. -A review of the state of practice of case study configurations and reporting. The review brings to light a very heterogeneous way of conducting and reporting case studies that precludes in-depth comparisons of the performance of approaches. -An extension of the taxonomy proposed by Turhan (2012) on ways that researchers can address cross-project problems based on the findings of our review.
The remainder of this article is structured as follows. In the next section, we present our systematic mapping study methodology. In Section 3, we give the foundations required for our mapping study. Afterwards, the results of our mapping study are presented in Section 4. and discussed in Section 5. Finally, we summarize our results and present our conclusions in Section 6.

Methodology
Our review follows the guidelines for systematic literature reviews proposed by Kitchenham and Charters (2007). In the following, we define our underlying research questions, inclusion and exclusion criteria, how we identified papers, and which data was collected for our study. We do not define our study as systematic literature review but as a mapping study, as no synthesis of the results was possibly due to the heterotenity of our findings.

Research Questions
The motiviation of this mapping study was answering two questions: 1) which techniques were already considered in the state-of-the-art of cross-project defect prediction; and 2) how well do the approaches perform? Ideally, we wanted to synthesize the findings to determine the best approaches from the literature. However, during our initial considerations of the literature, we realized that the case study settings are very diverse in terms of data, classifiers, and performance measures. This precludes concrete comparisons of performances between studies on a large scale. We would like to stress that this does not imply that the case studies were carried out in an improper way or results were not reported adequately. This was not the case for the publications we considered, the case studies on their own were well done and not problematic.
However, due to the diversity of the considered aspects, comparisons were almost impossible, meaning that considering all case studies as a whole is problematic. For example, if approach A is evaluated only on data set X and approach B is evaluated only data set Y, it is unclear how exactly A performs on Y and B performs on X. Due to this, only subsets of studies would be comparable to each other. Thus, we dropped the comparison of performance as aspect from this mapping study.
To study which techniques were already considered in the state of the art, we defined the following five research questions.
-RQ1. Which approaches were already considered for CPDP? -RQ2. Which classifiers were the most popular for CPDP studies? -RQ3. Which data sets were used within CPDP studies? -RQ4. Which performance metrics were used to assess CPDP? -RQ5. Which baselines were proposed approaches compared to?

Inclusion and Exclusion Criteria
To identify which papers should be part of our review, we defined the following criteria for inclusion: publications with a case study that includes CPDP experiments; or publications that discuss theoretical aspects of CPDP; or publications on tooling that specifically addresses CPDP; or fully automated defect prediction approaches that do not require any labelled data from the target product.
Additionally, we used the following exclusion criteria: publications that only address Within-Project Defect Prediction (WPDP) or only mention CPDP as aspect for future consideration; and publications that were not peer-reviewed; and publications that are not fully published in English.

Identification of Papers
We used Google Scholar for the identification of papers. The advantage of using Google Scholar in comparison to directly accessing the databases provided by, e.g., IEEE Xplore or ACM is that these databases are indexed by Google Scholar anyway. Moreover, Google Scholar searches can include results that are not contained by any of the standard indexes in case of publications at smaller conferences and workshops, especially if they are restricted to a certain region. Table 1 summarizes the terms we used for our search and the number of hits.
Due to the power of the search engine, we could not consider all results. Instead, we focused on the first 1,000 hits of the search for each query. Hence, we considered a total of 6,000 hits achieved. We have to note that huge parts  of the query results were overlapping, due to the semantic recognition by the search engine of the similar terminology. The titles of all hits were considered. In case the title somehow indicated a relation to defect prediction, the abstract and the actual publication were scanned for cross-project aspects. For this, we mainly considered the case study setup, but also the general structure of the approach. After this filtering, we selected the 49 publications that were included in our review. Moreover, we note that we found no results that were included after about hits 850 for any query, which gives us confidence that we did not miss any papers. Additionally, we checked the related work cited in each of the publications we included to provide a cross-check if we missed something. All identified papers were already covered by our Google Scholar search results.

Data Collection
Since we decided that we could not directly assess the quality of approaches realistically, due to the diversity in the case studies, we do not define quality check criteria as proposed by Kitchenham and Charters (2007). Instead, we define aspects that are important to describe the techniques that were used and how they were evaluated. Therefore, we collected the following data for each publication: -Approach. The approach towards CPDP that was considered in the publication. We summarize the approach of all publications in order to provide a good overview of the work that was done and, thereby, enable future research to utilize this collected body of knowledge. Moreover, to determine reporting guidelines, the work carried out so far must be taken into account to ensure that it can be addressed by the reporting guidelines. During our summary of the approaches, we did not always use the same mathematical formulations or wording as the authors. Instead, we harmonized the different representations of approaches into a common language and mathematical terminology. -Approach Type. Turhan (2012) (discussed in Section 4.13) provided a categorization of six ways to treat differences between data sets to improve the shifts between different project contexts: outlier detection; relevancy filtering; instance weighting; stratification; cost curves; and mixture models. As part of our review, we collect for each publication which of these type of techniques was proposed. In case a method does not fit the taxonomy proposed by Turhan, we use the category "other" and state in parenthesis what kind of approach it is. -Classifier. The machine learning or statistical classifier that was used in the work. We collect this data because many approaches considered in the literature do not define new classification models, but instead data treatments. Since the classification models can impact the results, it is important for consistent reporting and comparability to identify commonly used classification models. -Data. The data used in case studies has a major impact on the external validity of results and the comparability. If two approaches are evaluated on different data sets, it is unclear if similarities or differences in the results are due to the approaches or due to the data. -Case Study Setup. Important factors regarding the comparability of studies due to the case study setup are, e.g., how the data in a case study is used and to which baselines results are compared. -Performance Measures. Since different performance measures consider different aspects, comparisons between studies with different performance measures are very difficult. We collect information about the performance measures with the goal to identify a popular set of measures that covers all important aspects of prediction models which can be used for comparable studies. -Results. We summarize the results achieved as well as which numbers were reported as part of a publication. The summary of the results themselves provide good hints to researchers as to how techniques may perform when compared to each other, even though we were not able to create a general overall performance comparison through this review. We report the mean performance of the results if possible. Otherwise, we try to report other summarative statistics like median performance, if those are available.

Foundations
In this chapter, we introduce the foundations required for our mapping study. We start with a brief discussion of the terms WPDP and CPDP. Then, we formalize the problem of CPDP in order to be able to better discuss proposed approaches. Afterwards, we list the performance measures used in the literature. Finally, we give an overview about the publicly available defect prediction data sets, that were used in publications on CPDP.

Defect Prediction Types
Within this section, we want to clarify the terms WPDP and CPDP. WPDP is based only on data from the same project. Two kinds of WPDP are considered in the literature.
-Cross-validation studies, in which data from one revision of a product is split into partitions and one partition is used as test data and the other partitions are used as training data. -Studies in which previous revisions of the same software product are used to predict the defects in a newer revision, e.g., the usage of version 1.0 to predict defects in version 2.0.
In the WPDP setting, no data from other products is used to train the classifer. For CPDP we differentiate between Mixed-Project Defect Prediction (MPDP), mixed CPDP and strict CPDP. For MPDP, data from the target product is used together with data from other products. For mixed CPDP, no data from the target product itself is allowed, but data from older revisions. This data is used together with data from other products. For strict CPDP, only data from other products is allowed, old revisions from the target product may not be part of the training data.

Formalization of CPDP
Let S be the set of software entities s of which a software product 2 is comprised. Let M := {m 1 , . . . , m p } be a set of software metrics (Fenton and Pfleeger, 1997). For a software product S, the metrics assign a numerical value to all of its entities, i.e., for all entities s ∈ S. In CPDP, we have a set of software products S cand = {S 1 , . . . , S n } for which we know both the metric data as well as the concept, i.e., we have M (S), c S fully specified for each S ∈ S cand . We want to predict the defects of a software product S * , S * ∈ S cand , for which we know the metric data M (S * ), but not the concept c * := c S * . We call S * the target product and c * the target concept. In order to estimate the target concept c * , a hypothesis h : R p → {0, 1} is calculated.
Definition 1 (CPDP Learner). A CPDP learner takes as input the metric data M (S) and concepts c S for all S ∈ S cand and the metric data M (S * ) of the target product S * . The output of the training procedure is a concept h that estimates the target concept c * .
Please note that while S cand is the input for the CPDP learner, it should not be confused with the training data, which we denote with S train . The difference is that one popular approach for CPDP is relevancy filtering (see Section 4.11), which selects a subset of the candidate data as training data. Hence, S train ⊆ i=1,...,n S cand .
In addition to the above definitions, we use the following notations in the remainder of this paper.
-For convenience and better readability, we often simply write S cand instead of i=1,...,n S cand when we refer to relevancy filters. -Similarly, we write S cand instead of i=1,...,n S cand in the following.
char(m(S)) to denote the distributional characteristic c of the metric m for the software entities S, e.g., mean(m(S)) and max(m(S)).

Performance Measures
Most of the performance measures are based on the confusion matrix, i.e., the number of true positive (tp) predictions, false positive (fp) predictions, true negative (tn) predictions, and false negative (fn) predictions. The used measures based on these values are the following.
Moreover, two measures for successful predictions experiments based on the above are used.
succ 0.75 defined as the percentage of predictions that achieve recall > 0.75, precision > 0.75, and accuracy > 0.75. succ 0.7,0.5 defined as the precentage of predictions that achieve recall > 0.7 and precision > 0.5.
Moreover, two variantes of the Area Under the ROC Curve (AUC) are used. AUC is distributed between zero and one. The variants of AUC are defined using the Receiver Operating Characteristic (ROC).
-AUC uses the pf versus recall as ROC.
-AU C Alberg uses the recall and the percentage of modules considered to define the ROC (Ohlsson and Alberg, 1996).
As an alternative to AUC, the H-measure was proposed by Hand (2009). The key difference between AUC and H-measure is that AUC implicitly uses classifier-dependent misclassification costs, whereas the H-measure uses a prior distribution for costs.
Another measure that is used is the goodness of fit using the Hosmer-Lemeshow (HL) test (Hosmer and Lemesbow, 1980) and Huberts statistical procedure (Sharma, 1996). This way, two classification models are compared directly with each other and it is evaluated if the result is significantly different and how large the effect size, i.e., the difference between the results is.
Furthermore, measures related to the cost required for reviewing effort are used.
-N ECM = C1·f p+C2·f n tp+f p+tn+f n is the normalized expected cost of misclassification with C 1 the cost of a false positive (Type I error) and C 2 the cost of a false negative (Type II error).
-N ECM Cratio = f p+Cratio·f n tp+f p+tn+f n is another way to express the above, with C ratio = C2 C1 . In case this variant is used with a concrete ratio, e.g., C ratio = 10 we simply write N ECM 10 .
cost = s * ∈S * h(s * ) · LOC(s * ) with LOC(s * ) being the Lines Of Code (LOC) of the inspected entities. -N of B 20% , i.e., the number of bugs found when inspecting 20% of the code.
-N of C 80% , i.e., number of classes visited until 80% of the bugs are found.
Finally, one performance measure related to the consistency of results between experiments is used. Here, the aim is to evaluate how big the difference between classification results are. Using this intution, the following performance metric is defined (He et al, 2015).

Data Sets
Nine public data sets were used in the investigated publications on CPDP. We only give a general overview of the data, including the number of products, as well as the number and general types of metrics after the taxonomy by Fenton and Pfleeger (1997). Complete lists of metrics, including descriptions are provided in the publication or at the download location of the respective data sets.

NASA
This dataset was donated by NASA through the NASA Metrics Data Program (MDP). The data contains information about eleven products implemented in C++ and one product implemented in Java. The metrics measure static product metrics. 21 metrics are available for all twelve products, and depending on the product there are up to 39 metrics total. Additionally, the data contains for each module the information whether it was defective or not. The data is publicly available online 3 .

SOFTLAB
This dataset was donated by SOFTLAB, a Turkish software company. The data contains information about three software products in C. 30 static product metrics are available for each module of the products together with the information if the module was defective or not. The data is publicly available online 4 .

JURECZKO
This dataset was donated by Jureczko and Madeyski (2010). It consists of data about 48 product releases of 15 open source projects, 27 product releases of six proprietary projects and 17 academic products implemented by students, i.e., 92 released products in total. As metrics, they collected 20 static product metrics for Java classes, as well as the number of defects found in each class. The data is publicly available online 5 .

RELINK
This dataset was published by citeWu2011. The data contains defect information about three products. The defect labels were manually verified and not just automatically generated from SourceCode Management system (SCM) commit comments. 20 static product metrics are available for each module together with the information if an entity was defective or not. The data is publicly available online 6 .

AEEEM
This dataset was published by D'Ambros et al (2010). The data set contains information about five Java products. 61 software metrics are available, including static product metrics, process metrics like the number of previous defects, the entropy of code changes, and source code churn. The data is publicly available online 7 . An extension of this data with project factors was performed by Zhang et al (2014Zhang et al ( , 2015a, which is also publicly available online 8 .

MOCKUS
This data set is based on data collected by Mockus (2009). The data contains information about roughly 235,000 projects hosted on SourceForge and GoogleCode. From this huge body of projects, Zhang et al (2014Zhang et al ( , 2015a extracted 1,385 projects by filtering based on the programming language, projects with a small number of commits or short lifespan, and limited defect data or fix-inducing commits. The metrics obtained in the data set are 21 product metrics and 5 process metrics. The data is publicly available online 9 .

ECLIPSE
This data set was published by Zimmermann et al (2007). The data contains defect information about three Eclipse releases, collected on file and package level. 31 static product metrics are available on the file level, 40 metrics on the product level. The data contains information about the number of pre-and post-release defects for all measured entities. Additionally, metrics about the size of the Abstract Syntax Tree (AST) and the frequency of node types in the AST are available. The data is publicly available online 10 .

NETGENE
This data set was published by Herzig et al (2013). The data contains defect information about four open source projects that follow strict and industry like development processes. The data contains a total of 465 metrics, including static product metrics, network metrics, as well as genealogy metrics, i.e., metrics related to the history of a file, e.g., the number of authors or the average time between changes. The data is publicly available online 11 .

AUDI
This data set was published by Altinger et al (2015). The data contains defect information about three automotive projects developed by Audi Electronics Venture GmbH. The data contains a total of 17 static source code metrics. A special characteristic of this data set is, that the source code for which the metrics were measured was not written directly, but instead generated automatically from Matlab/Simulink models. Moreover, the generated source code follows the very strict MISRA coding guidelines (The Motor Industry Softw. Reliability Association, 2004). The data is publicly available online 12 .

Literature Review
In this section, we provide the review of the the state of the art of CPDP, ordered by the time of the publications. The publications and the information in this overview were collected according to the methodology discussed in Section 2. Some of the publications were first published at a conference and then invited to a journal to submit an extended version of their findings. We summarize both these publications together, due to the fact that the general approach, data, and case study setup are similar in both and that the conference publication is a subset of the journal extension.

Briand et al, 2002
Approach. ? if a model trained for one project (Xpose), is suitable for the prediction of another project (Jwriter). To this aim, they evaluate the ranking of defect prone-entities produced by the MARS model for Xpose for the Jwriter project.
Approach type. None. Classifier. Linear regression, MARS! (MARS!). Data. The authors consider 2 java products for which 15 static product metrics are measured. The data is not publicly available.
Case study setup. The authors trained the MARS based on the Xpose project and evaluated it based on the Jwriter project. The paper also performs other experiments, however, they are not in the cross-project context. Performance measures. benef it, precision, and recall.
Results. The authors do not report mean results for the performance measures, but report curves based on different thresholds that are used for the classification with the linear regression and MARS!. The results show that both the linear regression and MARS! are better than random guessing during the review and the the benef it 17.6 times the average costs of not finding and fixing a defect during the inspection of the source code.

Nagappan et al, 2006
Approach. Nagappan et al (2006) investigate which metrics are suitable to predict post-release defects. As part of this study, they also investigate the similarity between prediction models trained for different products. Concretely, they calculated whether the trained predicton models are correlated. They assume that if two predictors are correlated, they work for both products. Approach type. Other (metric type influence).
Classifier. Logistic Regression. Data. The authors consider 5 large-scale products from Microsoft for which 18 static product metrics are measured, mainly concerning complexity and size. The data is not publicly available.
Case study setup. The authors first performed Principle Component Analysis (PCA) (Pearson, 1901) to reduce the metrics to their principal components. Then, a logistic regression model was trained for each product. Using Spearman and Pearson correlation the authors tested if the coefficients of the models are correlated. 20 pair-wise comparisons between products were performed with two correlation measures, i.e., 40 model correlations were analyzed. Performance measures. Not applicable.
Results. For two products, both Spearman and Pearson correlation determined that the models are correlated. For one product only Spearman correlation was significant, for another only Pearson correlation. In total, only 6/40 correlations were significant, i.e., 15%. The authors conclude that CPDP is possible if the data comes from similar products.

Khoshgoftaar et al, 2008
Approach. Khoshgoftaar et al (2008) propose to use a combination of multiple classifers as well as data from multiple products for CPDP. They consider four different scenarios: (1) one classifier trained for one product; (2) multiple classifiers trained on a single product; (3) one classifier trained for multiple products; and (4) multiple classifiers trained on multiple products. In case multiple classifiers and/or products are used, majority voting is used to determine the classification. Approach type. Other (classification model  (2) and (4) with all classifiers are used at once. For (3) and (4) one product is used as target product and the other six for training. The authors utilize ANalysis Of VAriance (ANOVA) (Fisher, 1918)  Approach. Watanabe et al (2008) propose to compensate differences between products through a standardization technique that rescales the data. In a scenario with only one candidate product as training data, they propose to use this product as reference for the standardization of the target data. This shall increase the homogeneity between the target product and the candidate product. As formula for standardization, the authors propose to multiply each metric value of the target product with the mean value of the candidate product and divide this by the mean of the target product itself, i.e., for all s * ∈ S * . Approach type. Other (instance transformation). Classifier. C4.5 Decision Tree. Data. For both projects, data about three versions was mined. The mined data contained seven metrics and bug labels based on the comments of the SCM logs. The data is not publicly available.
Case study setup. The authors use two experiment configurations: (1) CPDP with their standardization; and (2) CPDP without their standardization. All versions of each project were used once as training data to predict the defects of all versions of the other projects. Performance measures. recall and precision.
Results. With the standardization (1), the authors report a mean performance 0.65 recall and 0.73 precision. Without standardization (2), the mean performance is 0.50 recall and 0.75 precision. Hence, they observed a mean gain of 0.15 in recall and 0.02 in precision in their experiments.

Turhan et al, 2009
Approach. Turhan et al (2009) propose to use training data which is similar to the target product. To this aim, they devised a method based on the k-NN algorithm for relevancy filtering of the training data from the union of all available candidate products S cand . Turhan et al measure the similarity between entities s, s with the Euclidean distance between the metric vectors of the entities, i.e., dist(M (s), M (s )). Based on the distance, they select for each entity s ∈ S * the k closest entities from the candidate data. Before the relevancy filter is applied, the logarithm is applied to all metric data taking pattern from Menzies et al (2007). Approach type. Relevancy Filtering.
Classifier. Naïve Bayes. Data. Seven products from the NASA data and all three products from the SOFTLAB data.
Case study setup. The authors used three experiment configurations: (1) WPDP with 90% of the target data for training and 10% as test data repeated 20 times; (2) CPDP with all data not from the target product as training data; and (3) CPDP with the k-NN relevancy filter on 10% of the target data as test data repeated 100 times. The NASA and the SOFTLAB data were not considered as a single data set but used in different experiments, i.e., no data from NASA was used to predict defects in the SOFTLAB data and vice versa. Hence, all experiments we report on here are performed once on the NASA data and once on the SOFTLAB data. Performance measures. recall and pf 13 .
Results. On the NASA data, the WPDP (1) achieves a median performance of 0.75 recall and 0.29 pf. The CPDP with all data (2) achieves a median 0.97 recall and with 0.64 pf. No summary statistics are reported for the complete data set for the CPDP with k-NN relevancy filter (3). Here, only results on a product level are reported. However, it can be seen in the results that the recall for CPDP with k-NN relevancy filter is consistently lower than that of CPDP with all other data, whereas for pf the opposite is the case. For the SOFTLAB data, the WPDP (1) achieves a median performance of 0.88 recall and 0.29 pf.
The CPDP with all data (2) achieves a median performance of 0.95 recall and 0.65 pf. For the CPDP with k-NN relevancy filter (3), no overall results for the complete SOFTLAB data are reported. However, based on the individual results for each product, the same conclusions as for the NASA data can be drawn.

Zimmermann et al, 2009
Approach. Zimmermann et al (2009) consider pair-wise CPDP without any data processing, i.e., one product is used to predict the defects of another product. Based on the performance of pair-wise predictions, Zimmermann et al propose to train a decision tree for the relevancy filter to determine the product best suitable for training. This decision tree is based on project factors (e.g., if a database is used) as well as the metric data. Project factors considered are, e.g., the organization that developed the code, whether internationalization was applied, or if the project uses a database. Factors from the training data itself are, e.g., the number of observations and the code churn. Approach type. Relevancy filtering.
Data. The authors consider twelve large-scale and well-known software products, e.g., Mozilla Firefox, Internet Explorer, the Windows kernel, and Apache Tomcat. For some of the projects, multiple versions are available, which means a total of 28 data sets are part of the case study. Six software metrics were measured for each data set. All of these metrics are designed in a way that they are relative with respect to the project size or number of commits, e.g., added LOC total LOC . The metrics cover code churn, pre-release defects, and code complexity. The data is not publicly available. Case study setup. The authors used two experiment configurations: (1) pairwise CPDP with all version except old versions of the same product (strict CPDP) and (2) a decision tree trained as described above to select training data. For both, the authors remove all trivial prediction models, i.e., models that only predict defect or no defect for instances. Performance measures. recall, precision, accuracy, and succ 0.75 . Results. Due to the removal of trivial models and old versions for predictions, the authors only consider 622 combinations of the pair-wise predictions, and not 28 · 27 = 756. The success rate of the pair-wise CPDP (1) was 0.034, i.e., only 21 of the 622 predictions achieved were successful according to succ 0.75 . The mean precision of the predictions is 0.374, recall and accuracy are not reported. With their decision tree procedure (2), they can raise the success rate for small areas of products. For example, they achieve a success rate of 0.324 in case the test product has more pre-release bugs than the training product, the test and training products are either implemented for different operating systems or both for Windows, and the standard deviation of the relative complexity is higher in the target product. Moreover, they analyzed the effect of project factors on the metrics in general. They observed that on the one hand, differences between project factors are often good for the recall, but bad for the accuracy of the models. On the other hand, having the same project factors is often good for the precision. Finally, they also analyzed the effect of difference in distributional characteristics of the numerical characterics used for the predictions on the results. Here, they observed that higher median values in the test data increase the precision and recall, but decrease the accuracy.

Camargo Cruz and Ochimizu, 2009
Approach. Camargo Cruz and Ochimizu (2009) propose to apply a power transformation Hoaglin et al (1983) to the metric data and then standardize it. The power transformation is based on the logarithm and the observation that software metrics, especially the size and complexity, often follow exponential distributions (Kan, 2003), which is the same as what Turhan et al (2009) do for the treatment of the data. The standardization of the data is based on the median. Concretely, the difference between the median value of a reference product S ref ∈ S cand ∪ S * and other products S. Together with the power transformation, the standardization formula iŝ for all s ∈ S, S ref . The authors consider only pair-wise CPDP and propose to use the training product as reference. Approach type. Other (instance transformation). Classifier. Logistic Regression. Data. The authors use defect data about classes from seven different Java products. For all products, three static product metrics are available. The defect data was obtained from both the SCM logs as well as the Issue Tracking System (ITS). Moreover, the authors did not use the defect label directly, but rather consider the most/least defective classes. For this, they used the median of the defects contained in the entire data set of a product as reference point. If a class contains less defects than the median, it belongs to the least defective classes, if it is greater or equal to the median, it belongs to the most defective classes. The data is not publicly available. Case study setup. The authors used two experiment configurations: (1) CPDP with their standardization; and (2) CPDP without their standardization. As training data, the authors only use one product of the data. As target products, two others are used. The other four products are not used for the evaluation of the defect prediction. Performance measures. HL test and Huberts statistical procedure. Results. Through the HL test the authors conclude that the CPDP model with standardization (1) has a better goodness of fit than the CPDP model without standardization (2). Through the Hubert statistic, they observe a small advantage due to the standardization for one of the target products and a larger advantage on the other product.

Jureczko and Madeyski, 2010
Approach. Jureczko and Madeyski (2010) proposed relevancy filtering through the clustering of products based on self-organizing maps 14 (Kohonen, 1982), as well as with k-means clustering with a value of k = 2, i.e., a separation of the data into two groups. For the training of the defect prediction model, the authors propose to use only data from the same cluster. Approach type. Relevancy filtering.
Classifier. Linear Regression. Data. All 92 products from the JURECZKO data. Case study setup. The authors used three experiment configurations: (1) CPDP with data from all other products; (2) CPDP with k-means clustering as relevancy filter; and (3) CPDP with self-organizing maps as relevancy filter. To test whether the cluster model is better than using all data, the authors use the Shapiro-Wilk test (Shapiro and Wilk, 1965). Performance measures. N of C 80% . Results. The authors report only the mean performance per cluster, the overall mean values are not reported. With the separation of the data into two clusters with the k-means clustering (2), the authors did not note any increase in prediction performance over using all data (1). Regardless of the aproach, about 47%-49% of the classes needed to be visited with the clusters and with using all data. With the self-organizing map (3), four clusters were created. However, the authors note that "there are releases that were classified into none of those clusters" (Jureczko and Madeyski, 2010). No information on how many of the products in the data could not be clustered is given. However, from the other data we believe that 58 products could be clustered and the remaining 34 products could not be clustered. For two of the four clusters that contain 26 of the products the performance of using the cluster is worse than using all data. For the other two clusters, containing 32 products, the performance is increased by 11% and 5% respectively.

Liu et al, 2010
Approach. Liu et al (2010) propose a CPDP classifier created through a genetic program. The genetic program generates an S-expression-tree (Rivest, 1997) to create an equation that models the defects using metric data. The authors propose three different ways to use the genetic program. The first is to use all candidate products S train = S cand to train the genetic program. The second is to use one of the candidate products S val := S i ∈ S cand for model selection and S train = S cand \ S val for training. This means, that multiple genetic programs are trained with S train and their performance is evalutated on S val . This is repeated for all possible candidate products once as S val . Then, the best of these models is used, i.e., the model that achieved the best performance on S val . The third approach is called validation-and-voting. The general strategy is the same as above, except that not only the best model on S val is used, but instead all models of these models are used, i.e., multiple genetic programs are used, one for each S val . The majority vote of these genetic programs defines the classification. However, all genetic programs that do not achieve a minimal performance criterion 15 are dropped and not allowed to vote. Approach type. Other (classification model).
Classifier. Genetic program (see above) as well as 1-NN, Alternating Decision Tree, Bagging, C4.5 Decision Tree, Decision Table Logistic Regression, k-NN Learning, Lines-of-Code, Locally Weighted Learning with Decision Stumps, Logistic Regression, Naïve Bayes, One Rule, Partial Decision Tree, Random Forest, Repeated Incremented Reduced Error Pruning, Ripple Down Rules, SVM, and Tree-Disc Classification Tree. Data. Seven products from the NASA data. Case study setup. The authors used four experiment configurations: (1) CPDP with the genetic program with all products for training; (2) CPDP with the genetic program and one product for model selection; (3) CPDP with genetic program and validation-and-voting; and (4) CPDP with each of the classifiers listed above other than the genetic program. To evaluate the statistical significance of the differences in performance, the authors use a standard t-test (Student, 1908). Performance measures. error, error T ypeI , error T ypeII , N ECM 15 , N ECM 20 , and N ECM 25 . Results. The authors report the values achieved on each product for the error, error T ypeI , error T ypeII and N ECM for configurations (1)-(3). For experiment configuration (4), only the mean values over all classifiers are reported for all performance metrics as well as the mean performance on all products in terms of error, error T ypeI , and error T ypeII each classifier individually. The genetic program trained with all products (1) achieves a mean performance of 0.337 error, 0.339 error T ypeI , 0.279 error T ypeII , 0.869 N ECM 15 , 1.029 N ECM 20 , and 1.247 N ECM 25 . The genetic program with one product for model selection (2)  Approach. An approach not only for CPDP, but rather for defect prediction in general and also effort prediction was first proposed by Menzies et al (2011) and later extended Menzies et al (2013). In their work, Menzies et al propose to first determine local regions of the data of high homogeneity before building a prediction model. Concretely, they use the WHERE algorithm to determine the homogeneous regions within the data. WHERE first applies the Fastmap algorithm (Faloutsos and Lin, 1995) to map the input data to two dimensions. The idea behind this is to have a fast approximation of a dimensionality reduction as achieved with PCA. Afterwards, the QuadTree clustering algorithm (Schikuta and Schikuta, 1993) is used to determine clusters within the data. After the QuadTree algorithm is terminated, a post-procedure to join small clusters is executed that determines the final clustering and, thereby, the homogeneous regions in the data. The clusters determined through the WHERE algorithm are then analyzed with the WHICH algorithm (Huang et al, 2010) to create rules for predictions. The rules learned with WHICH describe the non-defective data and, thereby, foster insights on attributes of non-defective data. Approach type. Mixture model. Classifier. WHICH algorithm, see above.
Data. Seven products from the JURECZKO data; the effort prediction part of the case study is out of scope of this review. Case study setup. The authors consider three experiment configurations: (1) rules created on all data; (2) MPDP data from the target product is allowed and (3) CPDP where no data from the target product is allowed. For configurations (2)-(3) all seven products in the data are merged into a single data set. Then, the data is clustered with WHERE. To learn rules with WHICH for a cluster C, the clostest cluster C 16 is used for training. In experiment configuration (2) all data from the cluster is used for training, for configuration, i.e., (3) data from the target product is removed from the training cluster. Performance measures. None of the performance measures introduced in Section 3.3 ist used. To measure the performance of the rules the authors measured the median, the interquartile range 17 and the worst case, i.e., the 100th percentile of the maximum of the defect values seen in the data set without applying the learned rule. Since the learned rules describe data that should not contain defects, this value should be low. Results. For the MPDP (2), the overall gain of using the local rules obtained after applying WHERE in comparison to global rules without clustering (1) in terms of the median of the defect values seen is about 13%, the interquartile range is decreased by about 71% and the worst-case performance is reduced by 66% 18 . For the CPDP (3), they observe a further gain in terms of the median of 60% over using all data (1). However, as the authors also note that this high percentage may be misleading due to the very small numbers involved, i.e., the absolute gain is only a reduction from eleven to four. This further decrease leads to an increasing interquartile range, which the authors interpret as a loss of stability of the results. The worst-case scenario is also reduced by an additional 70% for CPDP. 16 based on the median score of the dependent variable, for details, please consult the original article by Menzies et al (2013) directly. 17 called stabiliy in the article 18 In the paper, the authors report that the fraction of the sum of the medians of the cluster data by the sum of the medians of the global data is 0.64, i.e., a gain of not 13% but 36%. However, this takes the effort prediction rules into account. With only the defect prediction products, this value changes to 0.87, i.e., a gain of 13%. The same reporting differences hold true for the interquartile range and the worst case, where we also only report the results for the defect prediction data without the effort prediction results.

Turhan et al, 2011
Approach. Turhan et al (2011) suggest to create a MPDP model by augmenting WPDP training data with cross-project data, before training a classifer. In order to ensure that the cross-project data does not overwhelm the withinproject data, the authors suggest to create a family of classifiers trained with different amounts of cross-project data. Concretely, they suggest to first select candidate entities from other products using the k-NN relevancy filter suggested by Turhan et al (2009) (see Section 4.5). Then, k = 10, 20, 30, . . . instances are randomly drawn from the selected cross-project candidate data to augment the within-project data. For each of these augmented data sets, a WPDP classifier is trained and evaluated on data from the target product. The one with the best performance is then selected as classifier. Before training or relevancy filtering, the authors suggest to perform log-transformations for all metric data. Approach type. Relevancy filtering. Classifier. Naïve Bayes. Data. Seven products from the NASA data and all three products from the SOFTLAB data. Case study setup. The authors use two experiment configurations: (1) WPDP with 10x10 cross validation and (2) MPDP with 10x10 cross validation, where the training data is extended with cross-project data. Performance measures. recall, pf, and balance. Results. Using the Mann-Whitney-U test (Mann and Whitney, 1947) 19 , the authors determine that recall, pf, and balance are not statistically significantly different in both configurations for seven of the ten products used in the study. For one product, they noted an increase through the cross-project data augmentation in terms of balance, but a decrease in terms of pf, whereas recall was unchanged. For one product, they noted an increase in recall and balance, whereas pf was unchanged. For one product, all three performance measures were improved through the cross-project data. The authors conclude that mixing cross-project with within-project data can improve the results of predictions, but only to a limited degree and not reliably.

Premraj and Herzig, 2011
Approach. Premraj and Herzig (2011) conducted a study on the effect of network metrics on defect prediction models. Network metrics consider the underlying graph structure that connects software entities. In comparison to other product metrics, the network metrics take the interactions and information flow between software entities into account. Approach type. Other (metric type influence).
Classifier. k-NN, Logistic Regression, Naïve Bayes, Recursive Partitioning, SVM with a Radial Basis Function (RBF) kernel, and Tree Bagging. Data. In their study, the authors collect two versions of three open source Java projects, i.e., six products in total. The data contains nine product metrics, 25 network metrics, as well as information about the post-release defects. The authors published the data online, however, at this time we could not find a working internet address for the data. Case study setup. The authors used three experiment configurations 20 : (1) CPDP with only the product metrics; (2) CPDP with only the network metrics; and (3) CPDP with product and network metrics. Only the newest version of the three products was used for the CPDP context. The predictions were made pair-wise with all possible combinations of the three remaining products. Performance measures. recall, precision, and F-measure. Results. Premraj and Herzig determined that there is no significant difference between using product metrics (1) and network metrics (2) for CPDP.
No numerical values are reported, the evaluation is performed visually based on plots. Only for one combination of products they saw a difference for the precision, for the rest the results were almost the same. They did observe that using both product and network metrics together (3) decreases the performance of the prediction and, therefore, argue that the metrics used for predictions should be carefully selected and that more is not always better.

Turhan, 2012
Turhan (2012) created an overview of the problems due to data set shift for prediction models in software engineering, not only for CPDP, but also for cross-project effort prediction. The article does not contain CPDP techniques or a case study, which is why we break with the reporting pattern for this work.
Turhan identified six types of data set shift: (1) covariate shift, i.e., different distributions in the training and target data; (2) prior probability shift, i.e., the probabilities of the dependent variable differ (in case of CPDP, this means of the probability of defect is different in training and target data); (3) sample selection bias, i.e., problems due to a different nature in the training data and the target data, e.g., due to different maturity of the development process; (4) imbalanced data, i.e., an unequal number of defect-prone and non-defect-prone entities; (5) domain shift, i.e., problems that can arise if the same thing is done in a different manner, which can lead to inconsistencies; and (6) source component shift, i.e., when data from different contexts is merged, which can lead to too general data from which specific conclusions cannot be drawn.
Moreover, Turhan identified representative groups of techniques that can be applied to treat the above discussed problems. As broad groups, Turhan identified instance-based and distribution-based techniques. For each of these broad groups, Turhan identified three subgroups. For the instance-based techniques, the following subgroups were identified. (1) Outlier detection, i.e., the cleaning of the data through the removal of points that do not model the normal behavior but special conditions. Turhan notes that outliers may lead to covariate shifts in the data and, thereby, false generalizations. (2) Relevancy filtering, i.e., filtering the available training data based on the target data. The idea is to only keep the data for training, that is actually relevant for the target data. (3) Instance weighting, i.e., to use different weights for each instance in the training data, depending on the relevance of the instance for the result. In case this is based on the target data, instance weighting is a milder form of a relevancy filter, where instances are completely removed instead of only receiving a low weight. For the distribution-based techniques, the following subgroups were identified. (1) Stratification, i.e., avoiding probablitity shifts by sampling through certain guarantees that the underlying distribution did not change.
(2) Cost curves, i.e., decision support through a curve that describes predictor performances "over the full range of possible class distributions and misclassification costs" (Drummond and Holte, 2006). They can provide visual support for decision making and the analysis of predictor models.
(3) Mixture Models, i.e., using not a single model, but different models for different parts of the data, and thereby addressing the problem of source component shift.

Ma et al, 2012
Approach. Ma et al (2012) propose an instance weighting approach based on the idea of data gravitation. The general idea is to apply a weighing scheme to the training data based on the similarity with the target data. The authors take the term weight literally by adapting Newton's Universal Gravitation law (Newton, 1687) to calculate the force between software entities from the training data and the target data. The authors calculate the number of similar attributes simatts s of an entity s ∈ S as the number of metrics where the value of s is within the bound of the minimal and maximal value of the attribute in the target product: and They then calculate the weight of each training instance as Approach type. Instance weighting.
Data. Seven products from the NASA data and all three products from the SOFTLAB data.
Case study setup. The authors used three experiment configurations: (1) CPDP with 90% of the available training data; (2) CPDP with the k-NN relevancy filter proposed by Turhan et al (2009) (see Section 4.5); and (3) CPDP with the proposed weighting approach with 90% of the available training data. The products from the NASA data are used together as a training data. Using this data, the authors perform defect predictions on each of the product from the SOFTLAB data. Moreover, the authors once use each product from the NASA data as target product and the remaining products from the NASA data for training. Performance measures. recall, pf, F-measure, and AUC.
Results. The authors report the values for all performance metrics for the predictions on the SOFTLAB data and the F-measure and AUC for the predictions on the NASA data. On the SOFTLAB data, the CPDP without further treatment (1)  Only in terms of recall, the proposed weighting is outperfomed by the other two models. However, CPDP without further treatment (1) has mean recall of 1.0 and a mean AUC of 0.504, which indicates that the trained models degenerate to trivial predictions that always predict defects. Moreover, the recall of the weighting and the k-NN approach are very close to each other.

Peters and Menzies, 2012
Approach. A general problem in the way of CPDP is sharing data about proprietary products. Due to the potentially sensitive information that can be glanced from defect data, companies are unwilling to share such information. Peters et al (2013a) developed the approach MORPH to facilitate data sharing while ensuring privacy. The task of MORPH is to mutate the metric values of the entities s ∈ S with r ∈ [α, β] ∪ [−β, −α] and s N U N the nearest unlike neighbor of s, i.e., the NN of that has a different class label defined as The authors use the values α = 0.15 and β = 0.35. Approach type. Other (data privacy).
Data. Ten products of the JURECZKO data.
Case study setup. The case study has two goals. The first goal is to evaluate if the performance of the defect prediction suffers from the privatization of the data through MORPH. To this aim, the authors used two experiment configurations: (1) CPDP with each of the products in the data set once as target product and the other nine as training data; and (2) WPDP with 10x10 cross-validation. The defect prediction is performed with the original data as well as with privatized data. The second goal is to evaluate how well MORPH privatizes the data. To this aim, the authors used one more experiment configuration: (3) data swapping with p ∈ {10%, 20%, 40%} of the data swapped. Performance measures. F-measure for prediction performance. The privacy is evaluated by measuring how many queries to the privatized data, e.g., for certain subranges of attributes, return the same instances as to the original data.
Results. The authors do not report numerical results. Line graphs of the privacy and the F-measure for the different configurations and privacy methods are given instead. Regarding prediction performance, the graphs show only very small differences between the original (1) and the privatized data (2,3). Therefore, the authors conclude that MORPH and data swapping both do not degrade the prediction performance. Moreover, the privacy evaluation shows that MORPH (2) is comparable with swapping p = 40% of the data and better than swapping less data (3).

Z. He et al, 2012
Approach. He et al (2012) studied the general feasibility of CPDP, based on the findings of very low success by Zimmermann et al (2009). While Zimmermann et al only considered pair-wise predictions in their study, He et al consider combinations of one, two, or three products as training data. Combinations with more products are not taken into account due to the exponentially rising number of combinations. They disallow combinations with training data from the target product, i.e., other versions. Then, taking further pattern from the study by Zimmermann et al, the authors train a decision tree for the identification of training data that has a high chance for success. In comparison to Zimmermann et al, they base their decision tree solely on distributional characteristics. They use 16 distributional characteristics for each of the 20 metrics. Thus, they create a 320-dimensional vector for each combination of products they evaluated. As label for the learning, they use whether the combination of data was successful in terms of achieving high recall and precision or not.
Approach type. Relevancy filtering.
Data. 34 products of JURECZKO data.
Case study setup. The authors used three experiment configurations: (1) each product once as target product with all combinations of one, two, and three products as training data to determine the best possible combination for each product and the best possible result that can be achieved; (2) WPDP with 5x5 cross validation; (3) WPDP with data from old versions of the same target product; (4) the decision tree approach described above evaluated using 5x5 cross-validation with products as hold-out data during of the cross-validation.
Results. The best mean best-case CPDP performance in the experiment using all configurations (1) was achieved with the Decision Table. The mean performance over all target products is 0.735 recall, 0.560 precision, and 0.627 F-measure. For 18 of the 34 products at least one combination of training data that fulfills the criteria for a successful defect prediction was found, i.e., only succ 0.7,0.5 = 2.47% of all combinations led to a successful prediction. The best WPDP performance with 5x5 cross validation (2) is achieved with the C4.5 Decision Tree, with a mean performance of 0.486 recall, 0.558 precision, and 0.512 F-measure. For the WPDP based on old versions of the same projects (3), the best performance is also achieved by the C4.5 decision tree, with mean values of 0.585 for recall, 0.512 for precision, and 0.496 for F-measure. In a product-level comparison, they determined that for 18 products the best-case CPDP achieved with configuration (1) performed significantly better in terms of the recall and F-measure than the WPDP with cross validation (2). For precision, they observe a different effect, here the CPDP yields better results for four products, but is worse for ten products. The product-level comparison between CPDP (1) and WPDP with old versions (3) are similar for recall (14 times better CPDP better, one time worse) and F-measure (19 times CPDP better). For precision, CPDP is better for 11 products and lower for three products.
For their decision tree approach (4) the authors report a mean performance of 0.683 recall and 0.739 precision. The created decision tree is huge with 642 leaf nodes of which 270 lead to a successful prediction. Hence, the tree effectively contained 270 rules for good predictions. Through these results, the authors concluded that distributional characteristics can be used to determine good training data for CPDP.

Rahman et al, 2012
Approach. Rahman et al (2012) propose performance measures for the evaluation of CPDP. The proposed performance measures AUCEC and AUCECF are variants of AUC that take the cost effectiveness into account. The motivation for AUCEC /AUCECF is hence the same as for defect prediction in general. In practice, only parts of a software product are investigated in detail for quality assurance instead of the whole product. Assuming that a model ranks files by their defect-proneness, it becomes possible to inspect only few files and still find most of the defects. In case of ties in the ranking by a predictor, the authors propose to rank the smaller file higher, because less data needs to be inspected to find a defect with the same probability. AUCEC and AUCECF formalize this idea as a performance measure. AUCEC is measured using an ROC curve where the percentage of defects found is measured against the percentage of the LOC inspected. Similarly, the authors define AUCECF using the percentage of the number of files inspected instead of LOC. This is identical to the definition of AU C Alberg . Approach type. Cost Curves. Classifier. Logistic Regression. Data. The authors collected defect data for 38 versions of nine projects. They collected process metrics (e.g., number of commits; number of developers; added, deleted and changed LOC) and the LOC of files and combined the data with information about added features, improvements and defects from the ITS of the projects. The data is not publicly available. Case study setup. The authors use five experiment configurations: (1) with a threshold of 0.5 for the cutoff probability, i.e., all files predicted with a likelihood of at least 0.5 as defective are considered to be defective 21 ; (2) with the best possible threshold that maximizes the F-measure on the training data; (3) with the best possible threshold that maximes the F-measure on the target data; (4) WPDP based on old versions of the project; and (5) CPDP with a single metric as training data. Performance measures. F-measure, AUC, AUCEC, and AUCECF.

Results.
No numerical values are reported, instead visual analysis is performed supported with Mann-Whitney-U tests for statistical significance of the results. The case study shows that CPDP (1)-(3) performs worse in terms of F-measure, AUC, and AU C Alberg than WPDP (4). In terms of AUCEC, the authors do not see any significant differ between CPDP and WPDP. The single metric models show (5) that while the size metrics are effective if the performance is measured with AUC, they do not perform well using the cost sensitive AUCEC metric. From this, the authors infer that product metrics related to the size might be bad choices for prediction models in cost-sensitive settings.

Uchigaki et al, 2012
Approach. Uchigaki et al (2012) propose to build an ensemble model of single metric models. Concretely, they propose to create the ensemble as where h i is a Logistic Regression models trained only with metric m i . The weights w i are the contribution ratio of the sub-models to the classifcation as determined by the goodness of fit.
Moreover, the authors propose to preprocess the metric data with log transformation (see Section 4.5). Furthermore, the authors propose to standardize the data using z-score standardization, i.e., to transform all data as follows: Approach type. Other (classification model and instance standardization).
Classifier. Single-metric ensemble model (see above) and Logistic Regression. Data. All twelve products from the NASA data.
Case study setup. The authors use three experiment configurations: (1)  Approach. In an article by Canfora et al (2013) later extended in Canfora et al (2015), the authors propose a multi-objective approach for CPDP based on the findings by Rahman et al (2012) regarding cost effectiveness of models (see Section 4.17). Canfora et al consider two objectives: (1) maximization of the effectiveness of the defect prediction, i.e., to find as many defects as possible and (2) minimization of the cost of the reviewing effort due to the prediction. Hence, the objective functions are Since standard learning procedures only optimize the effectiveness of the defect prediction, but ignore the costs, the authors used a genetic program for the training of a classifier. Concretely, the authors interpret the hypothesis h representing the classifier as the fitness function of the genetic program. The structure of the classifier, e.g., the coefficients that define a regression model, are estimated using the genetic algorithm. The authors apply the vNSGA-II algorithm (Deb et al, 2002) to obtain a set of optimal solutions, i.e., a set of Pareto optimal (Coello et al, 2006) (7) a local model obtained with k-Means as clustering algorithm, with k = 10 determined by the Silhouette coefficient (Rousseeuw, 1987) and Logistic Regression as classifier; (8) a local model with MDS (Borg and Groenen, 2005) as clustering algorithm and association rules as classifier; (9) WPDP with 10x10 cross-validation and Logistic Regression as classifier; (10) WPDP with 10x10 cross-validation and a Decision Tree as classifier. The configurations (1)-(8) are using the data from all products, except the target product as training data. Performance measures. AUC for trivial model comparison (3)-(4), recall, precision, and review cost measured in KLOC for the other configurations. Please note that the authors use two different definitions for recall and AUC in the journal extension (Canfora et al, 2015): they once consider the commonly used recall of classes, i.e., how many classes that contain defects are found, and once the recall of the overall defects that were found. The difference is that the second accounts for classes that contain multiple defects.
Results. Since MODEP does not compute a single classifier, but a family of classifiers that is Pareto optimal in terms of costs and effectiveness, the authors decided to pair the results of MODEP with results that achieve the same effectiveness or costs achieved with other models. In the conference paper (Canfora et al, 2013), the authors pair based on effectiveness, i.e., for a given model that receives a certain recall, the authors pick the MODEP model that achieves the same recall. In the journal extension (Canfora et al, 2015), the authors pair based on the costs. Due to the large overall number of pairings considered, we do not report all mean values here 22 . Since the Logistic Regression yields better results than the Decision Tree, we only report on the pairings in which Logistic Regression is used. Moreover, the results for both recall and AUC are very similar independent of whether defective classes or number of defects are considered. Therefore, we report on the recall and AUC related to classes, because it is commonly used in other publications. Furthermore, the authors evaluated all configurations with and without z-score standardization.
Since the general conclusions hold for MODEP independent of the standardization, we leave out all non-standardized results, since the performance with standardization is generally higher. First, we consider the comparison with the trivial model. The Logistic Regression MODEP (1) achieves a mean AUC of 0.836, the trivial model with decreasing sorting (3) a mean AUC of 0.388, and the trivial model with increasing sorting (4) a mean AUC of 0.759. From this, the authors conclude that MODEP outperforms trivial approaches.
Second, we consider the pairings based on recall. For the comparison of the Logistic Regression MODEP (1) with the normal CPDP model (5) Approach. Peters et al (2013a) developed the approach CLIFF+MORPH to facilitate data sharing while ensuring privacy as an extension of the previously proposed MORPH (Peters and Menzies, 2012) (see Section 4.15). The CLIFF is a relevancy filter that uses the concept of power of subranges in the data. First, equal frequency binning is applied to create n = 10 subranges ranges(m i ) for each metric m i in the data 23 . The power is then calculated using the likelihood of a class based on the data in the bin, i.e., power(c|r) = (P (r|c) · P (c)) 2 P (r|c) · P (c) + P (r|not(c) · P (not(c)))) with the class label c ∈ {0, 1} and the range r ∈ ranges(m i ). Then, the overall power of an entity s ∈ S is calculated as the product of the power of the subranges in which the metrics m i fall, i.e., with r the range into which m i (s) falls within ranges(m i ). CLIFF then selects the p percent of data with the highest power. Once instances are selected with CLIFF, they are privatized with MORPH. Approach type. Relevancy filtering and other (data privacy).
Case study setup. The authors use five experiment configurations: (1) CPDP without privitization; (2) CPDP with MORPH; (3) CPDP with CLIFF+MORPH with p ∈ 10, 20, 40 percent of the data selected; (4) CPDP with data swapping (Reiss et al, 1982) with p ∈ {10, 20, 40} of the data swapped; and (5) CPDP with k-anonymity (Sweeney, 2002) with k ∈ {2, 4} indistinguishable members. For all of the above, the authors use each product once as target product and the other nine products for training. Performance measures. recall, pf, and G-measure for prediction performance. The privacy is evaluated by measuring how many queries to the privatized data, e.g., for certain subranges of attributes, return the same instances as to the original data.
Results. The values for recall, pd, and G-measure are reported for all ten products with all privitization approaches. We only report the results for the Naïve Bayes, as it performs best overall. However, the performance of k-NN is not statistically significantly different from Naïve Bayes. CPDP without privatization (1) achieves a mean performance of 0.236 recall, 0.091 pf, and 0.335 G-measure. CPDP with MORPH (2) achieves a mean performance of 0.229 recall, 0.089 pf, and 0.327 G-measure. CLIFF+MORPH (3) achieves the best performance with p = 40 percent of the data selected. The mean performance is 0.659 recall, 0.4 pf, and 0.598 G-measure. Data swapping (4) achieves the best performance with p = 10 percent of the data swapped. The mean performance is 0.225 recall, 0.084 pf, and 0.32 G-measure. k-anonymity achieves the best performance with k = 2. The mean performance is 0.195 recall, 0.08 pf, and 0.294 G-measure. In terms of privacy, CLIFF+MORPH (3) and k-anonymity (5) are almost perfect for all values of p and k with over 90% of queries failing. MORPH (2) and data swapping (4) are below 80%.

Peters et al, 2013b
This publication by Peters et al (2013b) was actually retracted due to problems with the case study setup 24 . However, we still list this work here for three reasons: (1) completeness of the review; (2) accurate statistics about classifiers, data usage, etc.; and (3) because the proposed approach is being used for comparisons regardless of the retraction (e.g., by Kawata et al (2015), discussed in Section 4.36). However, we only describe the approach suggested by Peters et al, the data used and the performance metrics. We do not elaborate on the case study setup or the results. Approach. Peters et al (2013b) propose a relevancy filter where the candidate data drives the filtering. For each entity in the candidate data s ∈ S cand the closest entity in the target data is selected. Through this, a subset of the target data is defined as Using S * closest , the same approach is used to determine the training data as subset of the candidate data as training data: Kocaguneli et al (2013) discuss the problem of experiments in a cross-project setting in general, and do not propose a concrete approach or conduct a case study, which is why we break with our reporting pattern. From a brief consideration of previously considered approaches for the cross-project context, not only for defect prediction, but also for effort prediction, the authors formulate three synergies between those approaches which they believe can advance the state of the art: (1) the synergy between supervised learning and transfer learning, which they state is already being explored with some success; (2) the synergy between semi-supervised learning and transfer learning, i.e., only labeling a small subset of the target product context manually and using transfer learning to augment the data; and (3) the combination of active learning, e.g., to reduce the available data to the most fitting the context before applying transfer learning.

Turhan et al, 2013
Approach. Turhan et al (2013) explore MPDP as an extension of their earlier work (Turhan et al, 2011) (see Section 4.11). The basic approach is identical to their previous work. However, they additionally explore how using less withinproject data affects the performance of predictors. Approach type. Relevancy filtering.
Classifier. Naïve Bayes. Data. Seven products from the NASA data, all three products from the SOFT-LAB data, and 63 products from the JURECZKO data.
Case study setup. The authors compare three configurations against each other: (1) WPDP with 10x10 cross validation and (2) WPDP with 10x10 cross validation, where the training data is extended with cross-project data; and (3) WPDP with only 10% of the within-project data for training augmented with cross-project data. The third configuration is only tested on a subset of 31 products of the JURECZKO data. The NASA and SOFTLAB data are used together as a single data set. The authors use a Mann-Whitney-U test to determine the statistical significance of their results. Performance measures. recall, pf, and balance.
Results. The authors report all three performance metrics for all products for the configurations (1) and (2). For configuration three, the results are only reported for a subset of the JURECZKO data. For the WPDP (1), the authors report a mean performance of 0.743 recall, 0.276 pf, and 0.710 balance on the NASA/SOFTLAB data and of 0.620 recall, 0.303 pf, and 0.612 balance on the JURECZKO data. For WPDP with 10x10 cross validation augmented with cross-project data (2), the authors report a mean performance of 0.784 recall, 0.275 precision, and 0.620 balance on the NASA/SOFTLAB data and of 0.675 recall, 0.305 precision, and 0.649 balance on the JURECZKO data. For WPDP with only 10% of within-project data augmented with cross-project data, the authors report a mean performance of 0.706 recall, 0.152 pf, and 0.671 balance on 31 products from the JURECZKO data. Using the Mann-Whitney-U test, the authors determine that augmenting the WPDP with cross-project data, improves balance in comparison to normal WPDP significantly in about half of the cases and does not decrease the performance in the other cases. With the reduced data from within the product, this finding still holds.

Singh et al, 2013
Approach. Singh et al (2013) do not propose any special approach, but evaluate pair-wise prediction without additional data treatment. Approach type. None. Classifier. C4.5 Decision Tree, Decision Table, k-NN, Naïve Bayes, Random Forest and SVM. Data. One Java product from the NASA data and one Java product mined by the authors, for which the data is not publicly available. Only Chidamber and Kemerer metrics were used from the available data.
Case study setup. The authors use two experiment configurations: (1) WPDP with 10x10 cross validation; and (2) pair-wise CPDP with each product once as training and once as test data. Performance measures. recall, precision, pf, and F-measure.
Results. The best overall performance is achieved by the decision tree. For WPDP (1), the authors report a mean performance of 0.690 recall, 0.650 precision, 0.302 pf, and 0.669 F-measure. For CPDP (2), the authors report a mean performance of 0.585 recall, 0.693 precision, 0.214 pf, and 0.624 F-measure.
The authors conclude that CPDP using Chidamber and Kemerer metrics is possible.

Herbold, 2013
Approach. Herbold (2013) propose a relevancy filter based on the clustering of products using distributional characteristics. For each product S, the characteristic vector defined as char(S) = (char 1 (m 1 (S)), . . . , char 1 (m p (S)), . . . , is calculated and the euclidean distances between these vectors are considered. While the authors state that any distributional charactistic can be used, they only use the mean value (i.e., char 1 = mean) and the standard deviation (i.e., char 2 = sd. The authors propose two approaches for CPDP based on this general idea: the first approach is to use the k-NN algorithm to select k products that have the most similar distributional characteristics. The second approach is to apply the EM clustering algorithm (Dempster et al, 1977) to create clusters of products based on the characteristic vectors of the candidate data joined with the target product. The number of clusters is determined automatically by the EM algorithm. The approach then selects those products as training data, that are within the same cluster as the target product. In case the target product is the only product in a cluster (i.e., in a cluster of size one), the EM algorithm is configured such that one less cluster than automatically determined is allowed and the clustering is repeated. Additionally, Herbold proposed a technique for the treatment of bias in the training data in case the distribution of the dependent variable, i.e., the defect-proneness is not roughly equal for defect-prone and non-defect-prone. With equal weighting, the entities of the training data are weighted such that the total weight of the defect-prone entities equals the total weight of the non-defect-prone entities, i.e., with w dp as weight for the defect-prone entities, w npd as weight for the nondefect-prone entities, and S train , c S train are the selected entities for training and the respective concept. The equal weighting is applied after the proposed training relevancy filtering techniques, because otherwise the property that both defect-prone and non-defect-prone entities have the same overall weight may be violated. Prior to the relevancy filtering, the metric data is standardized to the [0,1] using min-max normalization, i.e., for all s ∈ S. Approach type. Relevancy filtering, instance weighting, and other (instance standardization).
Case study setup. The authors used four experiment configurations: (1) WPDP with 10x10 cross validation; (2) CPDP with all data from other products; (3) CPDP with the k-NN product relevancy filter; and (4) CPDP with EM clustering for relevancy filtering. All of the above are applied with and without the equal weighting for bias treatment and for all seven classifiers listed above. For the CPDP all products are used as training data that are not other versions of the target product.
Performance measures. recall, precision, and succ 0.7,0.5 . Results. The equal weighting produces significantly better results consistently. Of the used classification schemes, the SVM with the RBF kernel performs best. Therefore, we only report those results in the following. The WPDP (1) achieves a mean performance of 0.57 recall, 0.63 precision, and 0.37 succ 0.7,0.5 . CPDP with all data (2) achieves a mean performance of 0.64 recall, 0.42 precision, and 0.09 succ 0.7,0.5 . The k-NN product relevancy filter (3) performs best for k = 20 with a mean performance of 0.74 recall, 0.40 precision, and 0.18 succ 0.7,0.5 . The EM clustering (4) achieves a mean performance of 0.71 recall, 0.40 precision, and 0.13 succ 0.7,0.5 . The author concludes that the proposed approaches outperform using all data, but are still worse than WPDP.

Z. He et al, 2013
Approach. He et al (2013) discuss if data obtained from open source software can be used to predict defects within proprietary software. Additionally, they suggest an approach for CPDP that consists of three parts: relevancy filtering of products, feature selection, and ensemble learning for classification. For relevancy filtering, their idea is to select products based on the separability measure between data sets proposed by Hido et al (2008). To measure the separability of two products S, S , 500 random samples S sample , S sample are drawn from both products 25 . Then, a logistic regression classifier is trained with the metric data M (S sample ), M (S sample ) as input and the data set membership as classification. The accuracy of this classifier is evaluated with 5x5 cross validation. Based on the accuracy, the distance between products is defined as dist sep (M (S sample ), M (S sample )) = 2 · (acc − 0.5).
The k closest neighbors according to this distance metric are used for training. The second part of their approach is feature selection. Following the suggestion by Hido et al (2008), the authors propose to filter the available metrics and only use those for training, which are similar in the training data and the target data. To measure the similarity they suggest to analyze the classification model trained to separate the data. The assumption is that the metrics with a high information gain (Mitchell, 1997) for the classifier are responsible for the differences between the training data and the target data. Therefore, the features with the highest information gain are removed to improve the homogeneity between the products.
The third part of their approach is to use an ensemble classifier (Rokach, 2010) and train a different classifer for the data of each product that was selected for training using the relevancy filter. The authors propose to use bagging (Breiman, 1996) for the ensemble learning, i.e., training a classifier for each product in the selected training data and using the majority vote of those classifiers for the overall classification.
Moreover, the authors propose to apply undersampling (Drummond and Holte, 2003) to the data to treat a bias towards non-defect-prone predictions due to the fact that more instances are non-defect-prone than defect-prone. In case there are more non-defect-prone instances than defect-prone instances, undersampling reduces the non-defect-prone instances through sampling from them until they have the same number as the defect-prone instances. Undersampling was also used successfully in context of WPDP, e.g., by Menzies et al (2008). Additionally, the data is standardized to the interval [0,1] using min-max normalization (see Section 4.25). Approach type. Relevancy filtering, stratification, and other (classification model, instance standardization, feature selection, and data source type influence). Classifier. Logistic Regression, Naïve Bayes, and Random Forest. Data. 34 products from the JURECZKO data. Performance measures. recall, pf, and G-measure. Results. No numerical results are reported, instead box-plots of the performance measures for the five models are shown. The boxplots show that their approach (5) slightly outperforms Turhan et al 's k-NN approach (4). They also performed a Mann-Whitney-U test between their suggested CPDP procedure and the k-NN approach. Here, they determined that in most cases their approach performs significantly better or equal to the k-NN approach, and only is worse for four, seven, respectively five proprietary products, for the Naïve Bayes, Logistic Regression, respectively Random Forest classifier. Regarding the number of metrics that should be removed by the feature selection, the authors determined that removing 60%-80% of the metrics yields the biggest improvement for CPDP. In terms of the overall performance, their suggested approach is slightly below the WPDP 5x5 cross-validation model (2). Both the NN approach and the approach suggest in this article clearly outperform using all data (4).

Nam et al, 2013
Approach. Nam et al (2013) propose to use Transfer Component Analysis (TCA) (Pan et al, 2011), a technique with the goal to learn a transformation φ that minimizes the distances between domains of the training data and target product, while maximizing the variance between the data. In their work, they use a linear transformation for φ. This linear transformation is then applied to each entry in the training data before training, thereby mapping the training data to the domain of the target data.
Moreover, the authors propose to standardize the data before applying TCA. They consider min-max normalization (see Section 4.25), z-score standardization (see Section 4.18), z-score-training standardization defined aŝ and z-score-target standardization defined aŝ Additionally, the authors propose TCA+. The aim of TCA+ is the automated selection of the best strategy for standardization before applying TCA, depending on how different data sets are. To determine the differences between data sets, the authors define the Dataset Characteristic Vector (DCV). The DCV consists of the mean, median, min, max, and standard deviation of the distances between the entities within the data sets. Based on the DCV, the authors define conditions, which state that if the distributional characteristics in the DCV are much more, more, slightly more, same, slightly less, less, or much less. TCA+ defines five rules that state standardization should be used: one for each of the four standardization approaches above and a fifth one that defines when no normalization shall be applied. Approach type. Other (instance standardization). Classifier. Logistic Regression. Data. All five products of the AEEEM data without context factors and all three products of the RELINK data. Case study setup. The authors used seven experiment configurations: (1) CPDP with no data transformations; (2) CPDP with TCA without standardization; (3) CPDP with TCA and min-max normalization; (4) CPDP with TCA and z-score standardization; (5) CPDP with TCA and z-score-training standardization; (6) CPDP with TCA and z-score-target standardization; (7) CPDP with TCA+; and (8) WPDP with 100 random 50:50 splits of the data. The CPDP was performed pair-wise between products of the same data set. Performance measures. F-measure. Results. The mean performances in terms of the F-measure over all pairwise CPDPs on the AEEEM are 0.31 without any transformation (1), 0.14 for TCA without standardization (2), 0.24 for TCA with min-max normalization (3), 0.41 for TCA with z-score standardization (4), 0.38 for TCA with z-scoretraining standardization (5), 0.38 for TCA with z-score-target standardization (6), and 0.41 with TCA+ (7). The WPDP (8) has a mean performance of 0.42. On the RELINK data, the mean performances are 0.49 without any transformation (1), 0.45 for TCA without standardization (2), 0.44 for TCA with min-max normalization (3), 0.57 for TCA with z-score standardization (4), 0.49 for TCA with z-score-training standardization (5), 0.59 for TCA with z-score-target standardization (6), and 0.61 with TCA+ (7). The WPDP (8) has a mean performance of 0.53. The authors conclude that the z-score standardization provides a significant improvement in comparison to no standardization. They note that overall, TCA+ performs a bit better than always using the same standardization and that the performance of TCA+ is comparable to the performance of the within-project predictions.

Panichella et al, 2014
Approach. Panichella et al (2014) propose to create an ensemble classifier from different base classifiers. Through a PCA, the authors determined that different classifiers can complement each other, because they are well-suited to detect defects in different parts of the data. Concretely, they propose to train a set of classifiers h 1 , . . . , h n for all S i ∈ S cand . Using the prediction results h i (s) ∀ s ∈ S i , i = 1, . . . , n as input, an COmbined DEfect Predictor (CODEP) classifier is trained, e.g., a Logistic Regression over the results of the classifiers h 1 , . . . , h n . Hence, the CODEP classifier is not trained directly on the available metric data, but only indirectly since it is based on results by classifiers trained on the metric data. Approach type. Other (classification model).

Classifier. CODEP with Alternating Decision Trees, Bayesian Networks, Decision Tables, Logistic Regresion, Multilayer Perceptron, and RBF Networks as classifiers h i and Logistic Regression and Bayesian Networks as CODEP classifiers.
Data. 10 products of the JURECZKO data. Case study setup. Within their case study, the authors compare the performance of using the six classifiers above by themselves to the performance of the two CODEP classifiers. The authors use all data except the data from the target product as training data. Approach. In an article by Zhang et al (2014), later extended in Zhang et al (2015a), the authors propose clustering based on the context factors of a product, e.g., the programming language, whether an ITS was used, and the number of developers in a product to perform rank transformation of the data. First, they create groups of products by binning the products according to their product factors. Then, they use Cliff's δ (Cliff, 1993), a measure for the effect size in case of stastical significant differences. If they determine a large effect size in the difference between two product groups, they are assigned to different clusters. If the difference is only small, they are merged into a single product group. This clustering is performed for every metric m i on its own and, consequently, the clusters are different for each metric. Once the clusters are created, the authors propose a rank transformation of the metric values. The transformation is based on the deciles of the metric values in a cluster as follows:m where dec j is the j-th decile of cl(S, i) for the product S and the metric m i . To train a defect prediction model they use the transformed metric values from all products. Approach type. Other (instance standardization and metric type influence). Classifier. Naïve Bayes. Data. All 1385 products of the MOCKUS data and all five products of the AEEEM data extended with context factors. Case study setup. The authors used nine experiment configurations: (1) WPDP with their rank transformation; (2) WPDP with log transformation (logarithm of all data is used, see Section 4.5); (3) CPDP with their rank transformation and only product metrics; (4) CPDP with their rank transformation and only product and process metrics; (5) CPDP with their rank transformation and only product metrics and context factors; (6) CPDP with their rank transformation and only process metrics and context factors; (7) CPDP with their rank transformation with all metrics and context factors; (8) CPDP with their rank transformation on foreign data; and (9) WPDP on the foreign data. For configurations (1)-(2), 10x10 cross valididation is used. For configurations (3)-(7) the authors do 10x10 cross validation with 10% of the products as hold-out data. The configurations (1)-(7) use only the MOCKUS data. The AEEEM data is used as foreign data for configurations (8) and (9). For configuration (8), the authors train the prediction model on the MOCKUS data and predict defects on the AEEEM data, configuration (9) is 10x10 cross validation on the AEEEM data. Performance measures. recall, precision, pf, F-measure, G-measure, MCC, and AUC.
Results. The authors report the mean values achieved by the configurations. The WPDP with rank transformation (1) has a mean performance of 0.580 recall, 0.525 precision, 0.359 pf, 0.534 F-measure, 0.521 G-measure, 0.214 MCC, and 0.615 AUC. The WPDP with log transformation (2) has a mean performance of 0.576 recall, 0.519 precision, 0.369 pf, 0.527 F-measure, 0.511 Gmeasure, 0.202 MCC, and 0.609 AUC. Due to the small and statistically not significant differences, the authors determine that rank transformation and log tranformation perform similar for WPDP. For the CPDP (3)- (7), the configuration with all metrics (7) yields the best results with a mean performance of 0.591 recall, 0.455 precision, 0.396 pf, 0.510 F-measure, 0.594 G-measure, 0.186 MCC, and 0.641 AUC. Overall, they conclude that they achieve a similar predictive performance as the WPDP model. For the evaluation with data from another data set (8) Approach. In an article by Fukushima et al (2014) and extended by Kamei et al Kamei et al (2015), the authors study how well CPDP is suited to the prediction of defective changes, also known as Just In Time (JIT) defect prediction. For the selection of similar products from a set of candidates for relevancy filtering, the authors propose a procedure based on the Spearmean correlation (Kendall, 1970). First, the three metrics m i1 , m i2 , m i3 with i 1 , i 2 , i 3 ∈ {1, . . . , d} with the highest correlation between a candidate product S and the target product are selected for each candidate product S ∈ S cand . Then, the pair-wise correlations within a product S are computed and used to define a three-dimensional internal correlation vector, i.e., intcor S = (cor(m i1 , m i2 ), cor(m i1 , m i3 ), cor(m i2 , m i3 )).
The euclidean distance between the internal correlation vectors for a candidate product and the target product is used to define the distance between products. The product with the smallest distance is used as training data, i.e., Furthermore, the authors explore the use of context factors for relevancy filtering, i.e., of the organization that develops the product, the intended audiance, the user interface, whether the product uses a database, and the programming languages. Using the context factors, the authors define a vector of binary variables, that define whether a context factor is fulfilled by a product or not, e.g., if a product is implemented by Eclipse, or if a product is written in C++. They then use the euclidean distance between these context vectors to define the similarity between products and propose to use the most similar for training.
In addition to using just the most similar product, the authors also propose to use the k most similar products together. Moreover, the authors propose a sampling-based approach that favors data from more similar products. They propose to sample 10−(r−1) 10 percent of the data of a product 26 , whereas r is the rank of a product according to their distance measure. Hence, all of the data of the most similar product is used, 90% of the data of the second most similar, and so on.
Additionally, the authors consider bagging for the prediction, i.e., training a separate model for each product in the data set, same as He et al (2013). In addition to normal bagging, where the vote of the classifier trained for each product is worth the same, the authors propose a weighted voting scheme, which is analogous to the sampling described above, i.e., the vote of the most similar products is counted fully, the vote of the second most similar product only to 90%, and so on.
For all of the above, the authors propose to use undersampling to deal with a potential bias in the data. Approach type. Relevancy filtering, stratification and other (classification model).
Classifier. Random Forest. Data. The authors use data about the changes of eleven open source largescale open source products. The data contains nine product and process metrics. The authors do not take the data as is, but perform an analysis for collinearity using Spearman's correlation coefficient. Due to this, the authors dropped one metric which was highly correlated and adopted the normalization approach from Nagappan et al (2006) to merge three correlated metrics into two uncorrelated metrics. This means that in total six metrics are used. The data is not publicly available online. Case study setup. The authors use 13 different setups experiment configurations: (1) WPDP with 10x10 cross-validation; (2) pair-wise CPDP between with all possible combinations for defect prediction; (3) CPDP with the most similar product using their Spearman correlation procedure (4) CPDP using the context factors as similarity measure; (5) CPDP using all data from other products for training; (6) CPDP using the three most similar products for training; (7) CPDP using the five most similar products for training; (8) CPDP using the similarity-based sampling approach proposed; (9) CPDP with rank transformations as proposed by Zhang et al, 2014 (see Section 4.29); (10) CPDP with bagging; (11) CPDP with bagging and the similarity-based voting scheme; (12) WPDP same as (1), but without undersampling the data; and (13) pair-wise CPDP same as (2), but without undersampling the data. Performance measures. recall, precision, F-measure, and AUC. Results. The authors report all values for all performance metrics for the WPDP with 10x10 cross validation (1) and the pair-wise CPDP (2). The WPDP (1) achieves a mean performance of 0.669 recall, 0.423 precision, 0.502 F-measure, and 0.778 AUC. The pair-wise CPDP (2), the achieves a mean performance of 0.585 recall, 0.353 precision, 0.404 F-measure, and 0.67 AUC. For the approaches (3)-(11) the exact results are not reported. Instead, bean plots of the percentage of AUC achieved in comparison to the WPDP are given. From their results, the authors conclude that similarity measured using context factors (4) yields better results than similarity measured using the Spearman correlation procedure (3), but both perform statistically significantly worse than within-project prediction. In contrast, the results when using multiple products together for training (5)-(8) are not significantly different from the WPDP results and, thereby, provide a strong advantage over simple CPDP. Moreover, the authors observe that simply using all products (5) is only rarely outperformed by the more sophisticated relevancy filters (6)-(8). Furthermore, they determined that rank transformations (9) do not perform well in the JIT setting. Additionally, they determined that the bagging-based approaches (10)-(11) do not perform statistically significantly different than WPDP and, thereby, provide an advantage over simple CPDP. For the evaluation of WPDP and pair-wise CPDP without undersampling (12)-(13), the authors report all values for all performance metrics. The WPDP without undersampling (12) achieves a mean performance of 0.303 recall, 0.645 precision, 0.385 F-measure, and 0.780 AUC. The pair-wise CPDP without undersampling (13) achieves a mean performance of 0.204 recall, 0.523 precision, 0.234 F-measure, and 0.7 AUC. From these results, the authors conclude that the undersampling does not impact the AUC, but makes a difference for recall and F-measure, which are both significantly better with undersampling.

Mizuno and Hirata, 2014
Approach. Mizuno and Hirata (2014) proposed a text-based approach for CPDP, inspired by spam filtering. Their first step is to separate the code lines from the comment lines. Moreover, the authors distinguish between end-of-line comments (e.g., //), block comments (e.g., /* */), and documentation comments (e.g., /** */). Then, they reduce the source code to its basic structure by removing all comments and whitespaces and replacing all identifiers and numbers with I, strings with S and characters with C. Each line and comment is then considered to be a token for the classification. Approach type. Other (classification model and metric type influence). Classifier. Text classification. Data. The authors use 25 products from the JURECZKO data and three products from the ECLIPSE data. Only the defect information of those data sets is used, the metrics were discarded and instead the source code was mined and tokenized by the authors themselves. The tokenized data is not publicly available.
Case study setup. The authors use three experiment configurations: (1) pair-wise CPDP between all products from different products; (2) CPDP with all products from different products as training data; (3) WPDP with all data from older versions of the product as training data. Performance measures. recall, precision, F-measure, accuracy. Results. The authors report the mean performance over all products per product for five of the eight products and, thereby for 21 of the 28 products. For the other seven products no results are reported. The mean performance of pair-wise CPDP (1) over these products is 0.298 recall, 0.581 precision, 0.310 F-measure, and 0.362 accuracy. For CPDP (2), the mean performance is 0.580 recall, 0.465 precision, 0.427 F-measure, and 0.656 accuracy. For WPDP (3), the mean performance is 0.431 recall, 0.527 precision, 0.349 F-measure, and 0.676 accuracy. The authors conclude that the CPDP is better than WPDP in terms of recall, but worse in terms of precision. Moreover, they determine that the source code tokens and end-of-line comment tokens are best suited for defect prediction.

Ryu et al, 2014
Approach. Ryu et al (2014) 27 propose to use data weighting in combination with boosting. For the data weighting, they propose to use the data gravitation approach by Ma et al (2012) (see Section 4.14). Within this work, the gravity weights are the similarity weights. They do not affect the classification or training process and are, hence, not related to the weights generated for boosting. Then, 50% of the training data are drawn randomly and 20% of the instances with the highest similarity weights are used as internal hold-out data for model validation. Once the weights are determined and the validation data is picked, the training loop starts. The first step of the loop is to resample the training data in order to treat a potential bias. The resampling is done through a combination of oversampling for data with high similarity weigths and undersampling for data with low similarity weights. Then, a SVM is trained using the remaining training data. Using AdaBoost (Freund and Schapire, 1997), the weights of all entities s ∈ S train that violate the classification are increased in order to get a classifer that does not repeat the mistakes. The overall classification is the weighted mean of the SVMs trained in each loop iteration. Prior to all of the above, the authors propose to use z-score standardization (see Section 4.18). Approach type. Instance weighting and other (classification model and instance standardization).
Classifier. SVM with a linear kernel which is internally used for the boosting; C4.5 Decision Tree, k-NN, Logistic Regression, Multilayer Perceptron, Naïve Bayes, PART Decision List, and Random Forest for comparison. Data. Seven products from the NASA data and all three products from the SOFTLAB data. Case study setup. The authors use four experiment configurations: (1) their approach; (2) a SVM with normal boosting; (3) Naïve Bayes with data gravitation as proposed by Ma et al (2012); and (4) the above classifiers as comparison to non-boosting techniques. As training data, the authors used all of the NASA products as training data for predictions of the SOFTLAB data. Moreover, the authors use each product in the NASA data once as target product with all other products from the NASA data for training. The SOFTLAB data is not used for the training of predictors. Performance measures. recall, pf, AUC, and H-measure. Results. The authors report the values for configurations (1), (2), and (4) for all performance metrics. For configuration (3) only AUC is fully reported, for recall and pf only the results on the SOFTLAB data are reported and no values for H-measure. Hence, we only summarize the result for AUC for configuration (3). For their approach (1), the authors report a mean performance of 0.581 recall, 0.306 pf, 0.753 AUC, and 0.335 H-measure. For the SVM with normal boosting (2), the authors report a mean performance of 0.337 recall, 0.095 pf, 0.744 AUC, and 0.332 H-measure. For Naïve Bayes with data gravition (3), the authors report a mean performance of 0.691 AUC. Of the classifiers used for comparison (4), Logistic Regression yields the best performance. The authors report a mean performance of 0.364 recall, 0.128 pf, 0.784 AUC, and 0.361 H-measure. Using the A-statistic (Vargha and Delaney, 2000;Arcuri and Briand, 2011) the authors compared their approach to the others. On the one hand, the authors determined that their approach is outperformed in terms of pf by the competitors. On the other hand, their approach outperforms all competitors in terms of recall. In terms of AUC and H-measure, their approach outperforms all others except Logistic Regression, which is similar.

P. He et al, 2015
Approach. He et al (2015) investigate strategies for the selection of metrics. To this aim, they define the T OP k metrics as the k most-often used metrics by trained defect predictors. For example, if a separate classifier is trained for each product in a data set, the authors count how often a metric m i ∈ M is used. This means that their approach can only be applied, after defect predictors were already trained and that their approach utilizes knowledge about the prediction results. In order to determine how many metrics should be selected, i.e., a good value for k, the authors introduce the notion of coverage where CF S S are the metrics selected by the Correlation-based Feature Subset (CFS) (Hall, 1998) approach. Additionally, the authors propose to optimize the T OP k by taking correlations between the metrics into account. Concretely, they determine the coverage for all subsets of T OP k and pick the one with the best coverage, such that it does not contain strongly correlated metrics, i.e.,

OP T T OP
with φ a threshold that defines when two metrics are strongly correlated. Furthermore, the authors propose to use log transformation for all metrics before creating prediction models. Approach type. Other (feature selection). Classifier. Bayesian Network, C4.5 Decision Tree, Decision Table, Logistic Regression, Naïve Bayes, and SVM with a polynomial kernel. Data. 34 products from the JURECZKO data. Case study setup. The authors use three experiment configurations: (1) WPDP with the latest version before the target product as training data; (2) WPDP with all prior versions of the target product as training data; (3) selection of the best subset of three products for CPDP using the brute-force strategy and knowledge about the prediction performance on the target product as proposed by He et al (He et al, 2012) (see Section 4.16). All three configurations were used with all classifiers, as well as all metrics, T OP 5 , OP T T OP 5 , and CF S S , max-relevance selction (Peng et al, 2005) and minimal-redundancymaximal-relevance (Peng et al, 2005). To determine OP T T OP 5 , the authors use φ = 0.6 as threshold for strong correlation. Performance measures. recall, precision, F-measure, succ 0.7,0.5 and consistency.
Results. The results are evaluated mainly visually using box-plots supported by the median of some results. As T OP 5 metrics of the JURECZKO data, the authors determine Coupling Between Objects (CBO), Response For a Class (RFC), Lack of COhession between Methods (LCOM), Efferent Coupling (Ce), and LOC (see Jureczko and Madeyski (2010)). The results of the case study show that the T OP 5 metric set is comparable to the full metric set, in the sense that it yields results that are within 90% of the original performance. This finding holds true for all three configurations and all classifiers. In comparison to the existing approaches, i.e., CF S S , max-relevance, and minimal-redundancy-maximal-relevance, the authors determine that T OP 5 yields a better prediction performance. Regarding the classifiers, Naïve Bayes is the best classifier for WPDP, i.e., configurations (1) and (2), Decision Table for CPDP, i.e., configuration (3). This finding holds true with and without the metric selection. Through their optimization, the authors determine CBO, LCOM, and LOC as OP T T OP 5 , because they are not strongly correlated and form the subset of T OP 5 with the best coverage. In terms of prediction performance OP T T OP 5 is determined to be close to T OP 5 in terms of precision, as well as predictions that are successfull according to succ 0.7,0.5 . For WPDP (1)-(2), T OP 5 outperforms OP T T OP 5 in terms of recall and F-measure. For CPDP (3), the differences between T OP 5 and OP T T OP 5 are not stastically significant. Using consistency and an ANOVA analysis, the authors determine that the T OP 5 metrics are consistent for different classifiers and not classifier specific.

Peters et al, 2015
Approach. Peters et al (2015) suggested LACE2 is an extension of CLIFF+MORPH (Peters et al, 2013a) (see Section 4.20) that accounts for multi-party data sharing instead of single party data sharing, i.e., multiple data owners take their turns to add data, then hand-off the data to the next data owner, so that they can add data. To facilitate sharing in a multi-owner environment, the authors propose to only add data, that is not yet represented in the data set with a strategy that adopts the Leader-Follower algorithm (Duda et al, 2012) called LeaF. To determine if an entity s ∈ S is already represented in the privatized data S privat , 100 entities S 100 ⊂ S privat are sampled. Then, the median of the distances to their nearest unlike neighbors (see Section 4.20) is calculated. An entity s is added to the data, if the distance to its nearest unlike neighbor is greater than the median of the 100 samples. The complete LACE2 approach first applies CLIFF for filtering, then LeaF to determine which new entities add information to the data, and finally MORPH to privatize the data. This procedure is repeated by each data owner in turn, to create the overall body of knowledge. Approach type. Relevancy filtering and other (data privacy). Classifier. k-NN. Data. Not applicable, since no experiment configurations addresses CPDP. Case study setup. The authors do not perform any experiments on the performance of CPDP, all executed scenarios in the case study consider only the WPDP scenario with cross-validation. Performance measures. Not applicable, since no experiment configuration addresses CPDP. Results. Not applicable for prediction performance. Regarding privacy, LACE2 outperforms CLIFF+MORPH, but both are on a very high level.

Chen et al, 2015
Approach. Chen et al (2015) propose an approach for MPDP called Double Transfer Boosting (DTB). They propose to apply k-NN filtering to select similar instances Turhan et al (2009) (see Section 4.5), and then to use oversampling with SMOTE (Chawla et al, 2002) to treat a potential bias in the data. Following that, they propose to determine weights for all instances using the data gravitation approach proposed by Ma et al (2012) (see Section 4.14). Finally, they propose to use transfer boosting based on TrAdaBoost (Dai et al, 2007). TrAdaBoost works with two data sets: a small data set from the same domain as the test data and a larger data set that might not be perfectly suited for the target domain. The boosting favors data from the small set by assigning higher weights, and assigns very low weights to data from the larger set that contradicts the current hypothesis, i.e., noisy data. In the considered scenario, the small data set are 10% of the data from the target product S * and the large data set are the candidate products S cand . Approach type. Instance weighting and other (classification model).
Classifier. Logistic Regression, Naïve Bayes, and Random Forest. Data. 15 products from the JURECZKO data.  (2012) (see Section 4.14); (6) WPDP with 10% of the data for training, the rest as test data; and (7) WPDP with 60% of the data for training, the rest as test data. Naïve Bayes is used as base classifier for configurations (1)-(5). For these configurations, the authors randomly draw 10% of the entities from the target product as within-project data for training and use all data from the other products as cross-project data. The remaining 90% of the target product are used as test data. This is repeated 20 times and mean values are considered. The same strategy for creating training and test data is also followed for (6) and (7), although 60% instead of 10% are drawn as training data for (7) and the authors use Logistic Regression, Naïve Bayes and Random Forest for these two configurations. Performance measures. recall, pf, G-measure, and MCC. Results. The authors report the results for all products and performance measures, as well as the mean performances. For their proposed approach (1), the authors report a mean performance of 0.702 recall, 0.330 pf, 0.664 Gmeasure, and 0.282 MCC. For CPDP without data treatment (2), the authors report a mean performance of 0.856 recall, 0.704 pf, 0.425 G-measure, and 0.126 MCC. For CPDP with the k-NN relevancy filter (3), the authors report a mean performance of 0.811 recall, 0.604 pf, 0.498 G-measure, and 0.167 MCC. For MPDP with the k-NN relevancy filter (4), the authors report a mean performance of 0.815 recall, 0.601 pf, 0.501 G-measure, and 0.169 MCC. For CPDP with data gravitation (4), the authors report a mean performance of 0.644 recall, of 0.350 pf, of 0.615 G-measure, and 0.226 MCC. Using the Mann-Whithney-U test, the authors determine that their approach is significantly better than the competitors in terms of the G-measure. In terms of MCC, their model outperforms all models except data gravitation, which is not statistically significantly different. For the comparison with normal WPDP, i.e., of their approach (1) with configurations (6) and (7), the authors report that their approach outperforms all classifiers when only 10% of the data is used for training (6). Moreover, the proposed approach outperforms Logistic Regression and Random Forest when 60% of the data is used (7) and is similar to Naïve Bayes.

Kawata et al, 2015
Approach. Kawata et al (2015) proposed a relevancy filter based on Density-Based Spatial Clustering (DBSCAN) (Ester et al, 1996). DBSCAN determines regions of high density in the data. Kawata et al propose to combine the metric data of the candidate products and the target product, i.e., M (S cand S * ) and then apply the DBSCAN algorithm to this data. As training data, all entities are selected that fall into the same cluster as any instance in the target data. Or vice versa: all data, that does not fall into the same cluster as at least one instance of the target product is discarded. Approach type. Relevancy filtering. Classifier. 1-NN, Logistic Regression Logistic Regression, Naïve Bayes, and Random Forests. Data. 56 products from the JURECZKO data. Case study setup. The authors use four experiment configurations: (1) CPDP with the DBSCAN relevancy filter as described above; (2) CPDP with the k-NN relevancy filter proposed by Turhan et al (2009) (see Section 4.5); (3) CPDP with the filtering approach proposed by Peters et al (2013b) (see Section 4.21); and (4) CPDP with no relevancy filtering. The authors used products with less than 100 entities as target products, and all products with more than 100 products together as candidate data. Performance measures. recall, precision, F-measure, G-measure, and AUC. Results. The authors report the median performance over all products for all results. Of the used classifiers, Naïve Bayes performs best. Therefore, we only repeat the results for this classifier in the following. The DBSCAN relevancy filter (1) achieves a median performance of 0.500 recall, 0.353 precision, 0.444 F-measure, 0.572 G-measure, and 0.624 AUC. The k-NN relevancy filter (2) achieves a median performance of 0.462 recall, 0.600 precision, 0.462 Fmeasure, 0.558 G-measure, and 0.624 AUC. The relevancy filter by Peters et al (3) achieves a median performance of 0.600 recall, 0.474 precision, 0.471 F-measure, 0.544 G-measure, and 0.612 AUC. CPDP without relevancy filtering (4) achieves a median performance of 0.077 recall, 0.600 precision, 0.143 F-measure, 0.143 G-measure, and 0.533 AUC. The authors conclude that their approach is comparable to the competitors.   Tables, Lo-gistic Regresion, Multilayer Perceptron, and RBF Networks; Bagging based on C4.5 Decision Trees; Bagging based on Naïve Bayes; Boosting based on C4.5 Decision Trees; Boosting based on Naïve Bayes; and Random Forest. Data. Ten products of the JURECZKO data. Case study setup. The authors use one experiement configuration in which they use all products for training except that target product. Using this configuration, the authors compare the performance of eight ensemble classifiers listed above. Approach. Amasaki et al (2015) propose to filter the metrics using the synonym pruning proposed for effort prediction (Kocaguneli et al, 2013). The general idea is to compute the distances between all pairs of entities for all metrics m i , i.e., dist(m i (s), m i (s )) ∀ s, s ∈ S with s = s . Then, all metrics are discarded, where no instance has the closest metric value to any other instance, i.e., only the metrics are kept. The authors propose to remove outliers using a similar procedure, i.e., to keep only instances for which any metric value is closest to any another instance, i.e., Moreover, the authors propose to use log transformation for all metrics before applying either of the above filters. Approach type. Outlier detection and other (feature selection).
Classifier. Logistic Regression, Naïve Bayes, Random Forest, and SVM with an RBF kernel. Data. 44 products from the JURECZKO data.
Case study setup. The authors combine the suggested metric and instance filtering with two existing CPDP approaches from the state of the art, i.e., the k-NN instance relevancy filter proposed by Turhan et al (2009) (see Section 4.5) and the k-NN product relevancy filter proposed by Herbold (2013) (see Section 4.25). They compare the performance of those techniques with and without their approach for data treatment. Hence, the authors use four experiment configurations: (1) k-NN instance relevancy filter with data treatment; (2) k-NN instance relevancy filter without data treatment; (3) k-NN project relevancy filter with data treatment; and (4) k-NN project relevancy filter without data treatment. Performance measures. recall, precision, F-measure, and AUC.
Results. The authors report the median values for all classifiers and both CPDP approaches with and without their data treatment. We restrict the reporting here to the result achieved with Logistic Regression, which performed best of the classifiers, although the SVM was a close second. For the k-NN instance relevancy filter with data treatement (1), the authors report a median performance of 0.661 recall, 0.374 precision, 0.489 F-measure, and 0.638 AUC. Without data treatment (2), the authors report median performance of 0.689 recall, 0.383 precision, 0.510 F-measure, and 0.662 AUC. From this, the authors conclude that their data treatment might actually lead to worse results if combined with the k-NN instance relevancy filter. For the k-NN product relevancy filter (3), the authors report median performance of 0.620 recall, 0.424 precision, 0.503 F-measure, and 0.660 AUC. Without data treatment (4), the authors report median performance of 0.625 recall, 0.376 precision, 0.494 F-measure, and 0.639 AUC. The authors note a statistically significant improvement of AUC using a Mann-Whitney-U test. From this, the authors conclude that there might be an improvement for this relevancy filter.

Ryu et al, 2015a
Approach. Ryu et al (2015a) proposed an extension of their earlier work on the application of boosting for CPDP (Ryu et al, 2014) (see Section Ryu et al (2015a)). The general approach is the same. There are two differences to their prior work. First, they do not assume that an SVM is used as internal classifier which is boosted, but allow any classifier. Second, the way the weights for boosting are calculated is changed, such that it can account for the costs of misclassifications. To this aim, they introduce a cost adjustment function β that penalizes misclassifications of data with different distributions. This is a variant of AdaCost (Fan et al, 1999), which penalizes all misclassifica-tions the same. Prior to all of the above, the authors propose to use z-score standardization (see Section 4.18). Approach type. Instance weighting and other (classification model and instance standardization).
Case study setup. The authors use six experiment configurations: (1) MPDP with their approach and Naïve Bayes as internally boosted algorithm; (2) CPDP with the k-NN relevancy filter proposed by Turhan et al (2009) (see Section 4.5) and Naïve Bayes as classifier; (3) CPDP with with SMOTE (Chawla et al, 2002) treatment and Naïve Bayes as classifier; (4) CPDP with Naïve Bayes without further data treatment; (5) CPDP with AdaCost; and (6) CPDP with TransferBoost. For all of the above, the authors choose each data set once as target product. As training data, the authors use the data of the remaining products. For configuration (1) the authors additionally use between 5%, 10%, and 25% of the data of the target product to enable their mixed-project approach. Performance measures. recall, pf, G-measure, and balance.
Results. The authors report the values for products, configurations and performance metrics, as well as the median performance achieved over all products. The performance between using different amounts of within-project data is similar, therefore, we summarize only the results for 5% within-project data here, because it is closest to strict CPDP. For their approach (1), the authors report a mean performance of 0.627 recall, 0.627 pf, 0.643 G-measure, and 0.633 balance. For Naïve Bayes with k-NN filtering (2), the authors report a mean performance of 0.736 recall, 0.528 pf, 0.560 G-measure, and 0.556 balance. For Naïve Bayes with SMOTE (3), the authors report a mean performance of 0.401 recall, 0.130 pf, 0.576 G-measure, and of 0.559 balance. For simple Naïve Bayes (4), the authors report a mean performance of 0.260 recall, 0.075 pf, 0.453 G-measure, and 0.470 balance. For AdaBoost (5), the authors report a mean performance of 0.527 recall, 0.199 pf, 0.625 G-measure, and 0.606 balance. For TransferBoost (6), the authors report a mean performance of 0.829 recall, 0.631 pf, 0.431 G-measure, and 0.460 balance. The authors use Mann-Whitney-U tests in combination with the A-statistic to evaluate differences between the configurations. The authors conclude that the cost-sensitive boosting is outperformed by TransferBoost and Naïve Bayes with k-NN relevancy filter in terms recall and by AdaCost, Naïve Bayes with and without SMOTE in terms of pf. However, using G-measure and balance, the proposed approach performs best among the configurations.

Ryu et al, 2015b
Approach. Ryu et al (2015) proposed a relevancy filter approach based on an adoption for string distances. The distance between two entities is defined as the number of different metric values after Raman and Ioerger (2003), i.e., dist(s, s ) = |{m i ∈ M : m i (s) = m i (s )}| (31) In the proposed approach, the authors first train a classifier on selected training data. For the relevancy filter, first outliers are removed using Mahalanobis distance (Mahalanobis, 1936). Then, all entities that are not in the neighborhood of an entity of the target product are removed, i.e., the training data is constructed as Using this data, the authors train a classifier. However, the classifier is not simply applied to all instances s * ∈ S * . Instead, the authors differentiate between three cases, following the LASER approach (Raman and Ioerger, 2003): (1) exactly one training instance has minimal distance; (2) multiple instances with the same classification have the minimal distance; and (3) multiple instances with different classifications have the minimal distance. In case (1), a neighborhood with distance is searched instead of only the nearest neighbors.
In case the neighbor now has multiple neighbors, the behavior is the same as for cases (2) and (3). If there is only one instance in the neighborhood, the classifier is applied. In case (2), the classification of the neighborhood is also used for the instance. In case (3) The results indicate that their proposed approach performs best overall in terms of balance.

Nam and Kim, 2015a
Approach. Nam and Kim (2015b) address the problem of combining different data sets, that may contain different sets of metrics. They consider a single candidate product S with metrics m 1 , . . . , m p and a target product S * with a different set of metrics m * 1 , . . . , m * p * . First, they propose to perform feature selection as proposed by Gao et al (2011) to reduce the number of metrics. Then, they suggest to calculate how well each metric in the candidate product is correlated to each metric in the target product, i.e., the pair-wise correlations cor(m i , m * j ) for i = 1, . . . , p and j = 1, . . . , p * . All pairs with a correlation higher than a certain cutoff threshold are kept as candidate metric pairs. Using maximum weigthed bipartite matching (Matouek and Gärtner, 2006), they select the candidate metric pairs that achieve the highest correlations without any duplicates in the metrics, i.e., no metric from the candidate product or target product occurs twice. The authors consider three different correlation measures: Spearman rank correlation; the p − value of a Kolmogorow-Smirnov test (Massey, 1951); and percentile-based matching defined as 9 k=1 percentileratio(10·k) 9 where percentileratio(n) = sp bp is the ratio of the n-th smaller percentile and the bigger percentile of the candidate and target product. Approach type. Other (data set combination). Classifier. Bayesian Networks, C4.5 Decision Trees, Logistic Model Trees, Logistic Regression, Random Forests, Simple Logistic model based on Logit-Boost, and SVM. Data. Five products from the NASA data, five products from the SOFT-LAB data, ten products from the JURECZKO data, all three products from the RELINK data, and all five products from the AEEEM data. In total, 28 products from five different data sets were used. Case study setup. The authors use four experiment configurations. (1) pairwise CPDP across data sets, i.e., all products from one data set were predicted with all products from the other four data sets; (2) WPDP using 2x2 crossvalidation 500 times repeated; (3) pair-wise 28 CPDP with the common metrics between data sets; and (4) pair-wise CPDP across data sets based on an approach proposed by He et al (2014) 29 for combining data from different data sets. The authors used the Mann-Whitney-U test to evaluate the statistical significance of results. This is done by using a Mann-Whitney-U test to check if the performance on the 1000 data splits was significantly different. Depending on the mean value, significant difference then means a win or a loss. In case of no significant difference of the results, the approaches tie. Moreover, they evaluated the percentage of pair-wise combinations, to which their matching based on correlations could be applied, since too low correlations between data sets mean that no matches will be found and, thereby, the approach cannot be applied.
Performance measures. AUC and win/tie/loss record, i.e., how often their approach outperforms others. Results. The authors note that the Kolmogorov-Smirnov test with a cutoff of 0.05 and Logistic Regression as classifier yields the best results. Therefore, the authors, and consequently this overview, only reports on the AUC performance achieved with that correlation measure. The authors report the median AUC for each product, each data set, and for all products together. We only report the aggregated results for all data sets, the performances on the each data set are similar to each other. The proposed Kolmogorov-Smirnov based matching (1) achieves a median AUC of 0.724, the WPDP (2) of 0.657, the CPDP with common metrics (3) of 0.636 and the CPDP based on the matching by He et al (2014) (4) of 0.555. The overall performance improvement was found to be statistically significant. Regarding the win/tie/loss record, the authors report 66.2% of wins for the Kolmogorov-Smirnov based matching (1) against WPDP (2), 66.2% of wins against CPDP using common metrics (3), and 82.0% of wins against CPDP with matching according to He et al (2014). However, this comparison is only done on 378 of the 600 pair-wise combinations, because no matching could be determined on the other 222 pairs. For AEEEM, 48% of the combinations were matches, for RELINK 88%, for JURECZKO 100%, for NASA 52%, and for SOFTLAB 100%.

Jing et al, 2015
Approach. Jing et al (2015) proposed an approach to create a Unified Metric Representation (UMR) which allows the usage of data accross data sets with different metrics. Assuming that M train are the metrics available for the training products and M * the metrics for the target product. Then  (Hardoon et al, 2004) to find linear transformations w train , w * for the training and test data that maximize the correlation between the two data sets. This is done by solving the eigenvalue problem where C train , C * , and C both are the co-variance matrices of the UMR of training data, target data, and both together. The defect prediction is then performed using the transformed data. Prior to all of the above, the authors propose to use z-score standardization (see Section 4.18). Approach type. Other (data set combination).
Data. Three products from the NASA data, all three products from the SOFTLAB data, all three products from the RELINK data, and all five products from the AEEEM data.
Case study setup. The authors use three experiment configurations: (1) CPDP with their approach; (2) CPDP with the k-NN relevancy filter proposed by Turhan et al (2009) (see Section 4.5); (3) CPDP with data gravitation propoposed by Ma et al (2012) (see Section 4.14); (4) CPDP with TCA+ proposed by Nam et al (2013) (discussed in Section 4.27); and (5) WPDP with 50% randomly sampled training data and the rest as test data 20 times repeated. The authors apply configuration (1) to all data. Configurations (2)-(4) are only used for data sets with overlapping metrics, i.e., if M both = ∅, which is the case for all pairings except the ones with the AEEEM data.
Configuration (5) is only used if there are no overlapping metrics, i.e., M both = ∅, which is the case for all pairings with the AEEEM data. For all CPDP approaches, the authors consider both pair-wise prediction with products from different data sets, as well as prediction with all data from one data set for products of other data sets. TCA+ is only used for pair-wise predictions. As classifier, the authors use k-NN for their approach (1) and Naïve Bayes for the others (2)-(5). The authors also report the performance of the other classifiers listed above for their approach for five predictions. Performance measures. recall, pf, F-measure, and MCC Results. The authors report nearly all values for products and configurations for recall, pf, and F-measure, as well as the mean values achieved for all predictions with the same number of common metrics 30 . The results achieved with all data are generally better than the results achieved with pair-wise predictions, which is why we restrict our summary to the results achieved with all data from a data set for training. This means we do not report the results of TCA+ (4), which are only available for pair-wise predictions. However, in general, TCA+ was somewhere between the k-NN (2) and data gravition (3) performance.
For the data with 28 overlapping metrics (NASA with SOFTLAB), the authors report for their approach (1) a mean performance of 0.81 recall, 0.02 pf, and 0.84 F-measure. For the k-NN relevancy filter (2), the authors report a mean performance of 0.65 recall, 0.09 pf, and 0.59 F-measure. For data gravitation (3), the authors report a mean performance of 0.74 recall, 0.37 pf, and of 0.44 F-measure. For the data with 3 overlapping metrics (NASA and SOFTLAB with RELINK), the authors report for their approach (1) a mean performance of 0.67 recall, 0.22 pf, and 0.55 F-measure. For the k-NN filter (2), the authors report a mean performance of 0.36 recall, 0.21 pf, and 0.34 F-measure. For data gravitation (3), the authors report a mean performance of 0.38 recall, 0.31 pf, and 0.30 F-measure. For the data with no overlapping metrics (NASA, SOFTLAB, and RELINK with AEEEM), the authors report for their approach (1) a mean performance of 0.62 recall, 0.17 pf, and 0.52 Fmeasure. For the WPDP (5), the authors report a mean performance of 0.70 recall, 0.27 pf, and 0.48 F-measure. The authors conclude that their approach outperforms all others, even WPDP in the settings where there are no common metrics between the data sets.
Moreover, the authors list the results for five prediction combinations achieved with the other classifers. k-NN performs best in the reported examples. For the same five examples, the authors report the MCC of their approach in comparison to others. Their approach always performs best, with one exception where it is beaten by the k-NN approach.

Cao et al, 2015
Approach. Cao et al (2015) proposed a transfer learning approach using neural networks. The first step of their approach is to remove outliers. All entities from both the training and target product are removed, where 80% of the metrics are more than the interquartile range outside of the upper or lower quartile, i.e., m i (s) i the upper and lower quartile and IQ i = Q 3 i − Q 1 i the interquartile range. Then, TCA (Pan et al, 2011) is applied, similar to the proposal by Nam et al (2013) (see Section 4.27). However, in comparison to Nam et al, Cao et al propose to use a kernelized TCA instead of a linear mapping. Then, a neural network is trained based on the mapped data. The neural network is trained in a way, that it takes a potential bias in the data to non-defect-proneness into account. Approach type. Outlier detection and other (instance standardization and classification model). Classifier. Logistic Regression and Neural Network (see above). Data. All three products from the RELINK data and all five products from the AEEEM data. Case study setup. The authors use four configurations 31 : (1) CPDP with their approach; (2) CPDP with TCA and Logistic Regression as classifier; (3) CPDP with TCA+ after Nam et al (2013); and (4) Logistic Regression without any data treatment. All configurations perform pair-wise CPDP between products from the same data set. The authors use WPDP to tune the parameters of their neural network for the CPDP. We do not report the WPDP results, because no information was given if and how the data was split. Performance measures. F-measure and AUC. Results. The authors report the F-measure for products and configurations, as well as the mean F-measure achieved over all products of the same data set. On the RELINK data, the authors report a mean F-measure of 0.66 for their approach (1), 0.59 for TCA with Logistic Regression (2), 0.61 for TCA+ (3), and 0.49 for Logistic Regression without data treatment. On the RELINK data, the authors report a mean F-measure of 0.41 for their approach (1), 0.41 for TCA with Logistic Regression (2), 0.41 for TCA+ (3), and 0.32 for Logistic Regression without data treatment. Values for AUC are not reported, only the ROC curves for some predictions are shown. The authors conclude that their approach outperforms the competition on the RELINK data, but not on the AEEEM data. They attribute this to the higher class imbalance of the AEEEM data.

Jureczko and Madeyski, 2015
Approach. Jureczko and Madeyski (2015) analyzed the impact of metrics on defect prediction results depending on different sources of the data, i.e., proprietary industrial products, open source products, and academic products. In order to analyze the importance of each metric for a linear regression defect prediction model, the authors define the importance factor with a i the coefficient in the linear regression model for the metric m i . Approach type. Other (metric type influence and data source type influence).
Classifier. Linear regression. Data. 83 products from the JURECZKO data.  (2) with N of C 80% of 55.38 and the academic model (3) with N of C 80% of 73.59. The only statistically significant difference is for the academic model, which is worse than the others.
The prediction of the open source products (5)-(8) performs best with a mean N of C 80% of 54.00 using the open source model (5), followed by using the industrial and academic model (8) with N of C 80% of 57.26, the industrial model (6) with N of C 80% of 57.67 and the academic model (7) with N of C 80% of 65.17. The only statistically significant difference is for the academic model, which is worse than the others.
The prediction of the academic products (9)-(12) performs best with a mean N of C 80% of 50.60 using the open source model (11), followed by using the industrial and open source model (12) with N of C 80% of 53.19, the academic model (9) with N of C 80% of 55.02 and the industrial model (10) with N of C 80% of 56.34. None of the differences in prediction performance are statistically significantly different.
Using the importance factor, the authors determine that the metric RFC is important in all the models based on all kinds of data. In case of academic and open source products, Afferent Coupling (Ca) and Ce are also very important. For industrial products LCOM and CBO are important.

Herbold, 2015
Herbold (2015) proposed a tool for the benchmarking of CPDP techniques. Since this is a pure tool paper which does not propose any approach or conduct a case study, we break with our reporting pattern. The proposed CrossPare tool provides a framework that models the general workflow of defect prediction experiments. CrossPare allows the definition of relevancy filters, data transformations, and weighting schemes. It is built around WEKA (Hall et al, 2009) and allows the internal usage of all features from WEKA. To define CPDP experiments, scripts are defined using an XML dialect.

Nam and Kim, 2015b
Approach. Nam and Kim (2015a) proposed with CLAMI a technique for unsupervised defect prediction. While this is not a defect prediction technique, it fits our inclusion criteria since it is a fully automated technique that does not require labeled data from within the project. CLAMI actually consists of two parts, Clustering and LAbeling (CLA) and Metric and Instances selection (MI).
CLA clusters the entities based on how many metric values are greater than the median, i.e., for s ∈ S. clust(s) can also be interpreted as an integer. The median of these cluster integers is used to define the labeling of entities. All entities whose cluster integer is higher than the median cluster integer are labeled as defective, i.e., c CLA (s) = 1 if clust(s) > median(clust(S)) 0 otherwise.
MI selects metrics and entities for the training, that do not violate the above defined classification and, thereby, removes noise from the data. Only the metrics with the fewest violations of the classification, i.e., that minimize are selected. Then, all entities that still violate the classification are removed, i.e., Approach type. Outlier detection and other (feature selection and unsupervised learning).
Classifier. Bayesian Networks, C4.5 Decision Trees, Logistic Model Trees, Logistic Regression, Random Forests, Simple Logistic model based on Logit-Boost, and SVM. Data. All three products from the RELINK data and all four products from the NETGENE data.
Case study setup. The foundation of the case study setup are 500 2x2 cross validations data splits, through which the authors define 1000 data splits with 50% of the data. Based on these splits, the authors use five experiment configurations: (1) CLA only applied to a split; (2) CLAMI applied to a split; (3) WPDP with the counterpart of the split used for training and the split itself for validation, i.e., the normal 2x2 cross validation; (4) comparison to a threshold-based technique adopted from Marinescu (2004) and ; and (5) an expert-based approach proposed by Zhong et al (2004). All of the above are performed with a Logistic Regression classifier. Only CLAMI (configuration 2) is also performed with the other classifiers. To evaluate the differences between the classifiers, the authors use the Friedman test with the Nemeyi test as post-hoc test (Demšar, 2006) when they compare multiple models and the Mann-Whitney-U test when performing pairwise comparisons. Performance measures. recall, precision, F-measure, and AUC. Moreover, the authors consider the mean rank of the five configurations achieved with a metric.
Results. The authors report the mean results of the performance metrics over the 1000 data splits for all products. AUC is not reported for configurations (1), (4), and (5). For CLA (1), the authors report a mean performance of 0.692 recall, 0.594 precision, and 0.630 F-measure. For CLAMI (2), the authors report a mean performance of 0.709 recall, 0.595 precision, 0.636 F-measure, and 0.724 AUC. For WPDP (3), the authors report a mean performance of 0.569 recall, 0.616 precision, 0.584 F-measure, and 0.694 AUC. For the thresholdbased approach (4), the authors report a mean performance of 0.260 recall, 0.749 precision, and 0.251 F-measure. For the expert-based approach (5), the authors report a mean performance of 0.606 recall, 0.773 precision, and 0.647 F-measure. Regarding the rankings of the technique, CLA and CLAMI are quite similar. They are the best ranked techniques in terms of recall with mean ranks of 2.143 for CLA and 2.000 for CLAMI. However, they are the worst ranked techniques in terms of precision with mean ranks of 3.71 for CLA and 3.857 for CLAMI. The good rankings in terms of recall are sufficient to reach the second and third ranks in terms of F-measure with 2.357 for CLA and 2.429 for CLAMI, only beaten by the expert-based approach with a mean rank of 1.929. The authors conclude that CLA and CLAMI can compete even with WPDP performance and the expert-based approach and only have drawbacks in terms of precision. The comparison of different classifiers for CLAMI determined that Naïve Bayes and Logistic Model Trees perform statistically significantly better than SVMs and C4.5 Decision Trees, both in terms of AUC and F-measure. The Random Forest, Bayesian Network and Logistic Regression are not signficantly different from the other classifiers and, thereby, form a mid-field.

Altinger et al, 2015
Approach. Altinger et al (2015) evalute how well CPDP works in the automotive domain. In the concrete setting, very restrictive development processes with automatically generated code from models, as well as strict coding guidelines were considered. The assumption of the authors was that CPDP should work well in this setting, since the restrictive setting should lead to homogeneous source code. The authors propose to use undersampling and test different data normalization techniques, as well as a relevancy filter from the literature. Approach type. Relevancy filtering and other (instance standardization). Classifier. SVM with RBF kernel. Data. Two products from the AUDI data. Case study setup. The authors use six experiment configurations: (1) WPDP where the data from the first 50% of versions is used for training and the rest for evaluation; (2) CPDP with k-NN relevancy filter proposed by Turhan et al (2009) (see Section 4.5); (3) CPDP with min-max normalized data (see Section 4.25); (4) CPDP with z-score standardized data (see Section 4.18); (5) CPDP with z-score standardized data based on the target product after Nam et al (2013) (see Section 4.27); (6) CPDP with data standardization after Watanabe et al (2008) (see Section 4.4). Since only two products are used, all CPDPs are pair-wise between the products. Performance measures. recall, precision, and F-measure.
Results. The authors report the values for all products, configurations, and performance metrics. With WPDP (1), the authors report a mean performance of 0.740 recall, 0.185 precision, and 0.295 F-measure. For CPDP with the k-NN relevancy filter (2), the authors report a mean performance of 0.615 recall, 0.195 precision, and 0.300 F-measure. For CPDP with min-max normalization, and both z-score standardizations (3)-(5) the authors report a mean performance of 0.615 recall, 0.180 precision, and 0.275 F-measure. For standardization after Watanabe et al (6), the authors report a mean performance of 0.585 recall, 0.210 precision and 0.315 F-measure. The authors conclude that the overall performance of all models is lacking. Due to this, the authors further investigated the cause for this and determined that the correlation between the metrics and the defect information is weak and, moreover, the information gain due to the metrics is rather low. Hence, the metrics do not seem to contain the required information about the defects. Moreover, the authors performed PCA to visually analyze if the defective regions in the data are overlapping, which is a requirement for CPDP. Here, the authors determined that the defective regions were partially disjunctive, which also explains the weak performance.

Discussion
Using the data collected from the literature review, we now answer our research questions and comment on the complexity of case study reporting.

Research Questions
RQ1: Which approaches were already considered for CPDP? Figure 1 gives an overview of the approaches considered in the state of the art. The taxonomy proposed by Turhan et al (2011) does not cover all considered approaches. We identified seven types of approaches that extend the existing taxonomy: instance standardization of the training and/or test data according to specific rules with the intention to reduce differences between projects; classification models which are tailored for CPDP; feature selection to determine a subset of metrics best suited for CPDP; data privacy without inhibiting the prediction performance; metric type influence on CPDP performance; data source type influence, e.g., the how open source products influence the prediction of proprietary product; and data set combination with none or partially overlapping features.
Among the approach types, relevancy filtering, classification models, and instance standardization are the topics that gained most attention with 12-14  publications each. The topic instance weighting, metric type influence, feature selection, stratification, cost curves, data privacy, and outlier detection received mild attention with 3-5 publications each. Influence of data source types and data set combination are only seldomly studied with two publications each topic. However, we note that the first publications on data set combination was in August 2015 indicating that this is a new topic. Mixture models were nearly ignored as a potential solution, with only one work focusing on them.
RQ2: Which base classifiers were the most popular for CPDP studies? Figure 2 gives an overview on the classifiers used in the publications. 38 different base classifiers were considered in total. The two most popular are Logistic Regression and Naïve Bayes, which were both used in 24-25 publications, followed by Random Forest, SVM, and C4.5 Decision Tree with 11-16 publications. k-NN, Bayesian Network, Decision Table, Linear Regression, Multilayer Perceptron, 1-NN, Alternating Decision Tree, and Bagging received mild attention with 3-7 publications considering them. All other models, including those proposed within the literature specifically for CPDP were only used at most two times. Regarding which classifiers are best, the literature is inconsistent. As candidates, Naïve Bayes, Logistic Regression, Random Forest, SVM, k-NN, as well as all algorithms specifically proposed for CPDP should be considered based on the results reported in the publications.  Figure 3 gives an overview on the usage of the data sets. For the JURECZKO data, the figure does not show the overall usage of the data, but a more finegrained look where less than or equal to 10 products, 11-20 products, 21-30 products, 31-40 products, 40-50 products and more than 50 products are used. This is to account for our observation that different subsets of the data were used within the literature and we think that the size can be used as relevant indicator to differentiate between the subsets.
The JURECZKO data was the most popular in the literature and was used in 20 publications. However, most publications used subsets of different size. Seven of these publications used less than or equal to 10 products, only five used more than 50 products. The second most popular data set is the NASA data which was used twelve times, followed by SOFTLAB, which was used seven times. The newer datasets RELINK and AEEEM were used five times each and gained traction in the later publications, but were, overall, used less than the JURECZKO, NASA, and SOFTLAB data. The other newer data sets MOCKUS, NETGENE, and AUDI were all used in one publication each. The ECLIPSE data was only used once, even though it has been available longer than all other data sets except NASA. Additionally, one data set should have been public, but the link was dead. Data that is not public was used by eight publications.
RQ4: Which performance metrics were used to assess CPDP? Figure 4 gives an overview on the performance metrics that were used within case studies. We distinguish between two types of metrics: overall performance metrics (Figure 4(a)) and metrics that consider specific aspects (Figure 4(b)). Of the metrics that consider the overall performance, F-measure is the most popular and used in 19 publications, followed by AUC, which is used in twelve publications, and G-measure, which is used in seven publications. The other measures received only mild or minor attention and were used in at most four publications. However, we note that costs were considered in eight publications, although with different metrics.
Of the aspect-specific metrics, recall was by far the most important metric and used in 28 of the publications, followed by precision and pf which were in 17, resp. 14 publications and are almost equally popular. The other metrics were each only used two times.
Interestingly, F-measure which is the harmonic mean of recall and precision was used more often than G-measure, which is the harmonic mean of recall and 1 − pf , while precision and pf were used almost equally often.
RQ5: To which baselines were proposed approaches compared? Figure 5 gives an overview on the baselines against which proposed methods were compared. We distinguish between general baselines, i.e., WPDP are CPDP without any specific approach and approaches that were previously proposed in a publication on CPDP. As general baseline, WPDP with crossvalidation is the most popular and used in 20 publications, followed by CPDP with all non-target product data as training data used in 17 publications and pair-wise CPDP between the products used in twelve publications and WPDP with old versions of the same project used in four publications. LOC ranking of modules was used only once.
Of the approaches previously proposed for CPDP, the k-NN relevancy filter proposed by Turhan et al (2009) (discussed in Section 4.5), is used 9 times as baseline and, thereby, somewhat regularly. Nine more approaches were also used between 1-3 times as baseline.

Lessons learned
Our mapping study showed that the work carried out for CPDP is very diverse. While this is, in general, a positive points because many different techniques, data sets, and performance measures are being considered, it makes comparisons between approaches almost infeasible in form of a metastudy that synthesizes the results. The following points are especially problematic in that regard: -CPDP studies are only comparable, if the exact same data is used. Due to different data sets that are nowadays available, many publications are 12 7 4 4 3 3 3 2 2 2 2 1 1 1  based on different data sets, the largest subset of publications that uses the exact same data we found were six. Papers that use multiple data sets often have a difference of more than 10% in prediction performance between the data sets. Even if only a subset is used, one cannot compare the prediction results just on the subset, because the training data used would still be different. Hence, one cannot just assume that for publications A and B with studies based on data(A) = data(B), that approach A outperforms approach B if perf ormance(A) > perf ormance(B). Only conjectures are possible, which would impose a major threat to validity of the conclusion.  -The diverse usage of performance metrics makes comparisons hard. While recall seems to be a generally accepted metric for CPDP studies, it only covers one aspect of the evaluation, i.e., if bugs are found. For all other metrics, there seems to be a strong disagreement within the community with at least four competing parties: 1) those in favor of the F-measure and precision in addition to recall ; 2) those in favor of the G-measure with pf in addition to recall ; 3) those in favor of the threshold-free AUC ; and 4) those in favor of cost-sensitive metrics. Studies based on different metrics are hard to compare. To some degree, a later inference of the confusion matrices is possible (Bowes et al, , 2013 and, thereby, the calculation of metrics that are not reported. However, this is not possible with all metrics, e.g., cost sensitive metrics and AUC cannot be calculated that way.
In addition to these problems with comparability, we also observed a general lack of replication of previously proposed techniques. Only ten of the approaches proposed in the state of the art were ever re-used in a case study by another publication, the other approaches were, apparently, never repli-cated as part of another public case study, which in itself is a major threat to the external validity of the results.

Conclusion
In this article, we provided a systematic mapping study off the state-of-theart of CPDP. Our review detected 49 publications that address the topic of CPDP. We systematically summarized the contributions of all publications, including the approach proposed, information about the case study setup, and the results achieved. Through our findings, we were able to extend the taxonomy by Turhan (2012) on how cross-project problems can be addressed. Moreover, we detected problems regarding the comparability of CPDP results across publications due to a diverse usage of data sets, performance metrics, and base classifiers combined with a lack of replication of previously proposed approaches for comparison.