{'fraction_non_alphanumeric': 0.026490369218099465, 'fraction_numerical': 0.006817038019856339, 'mean_word_length': 4.853776111408677, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
New developer metrics: Are comments as crucial as code contributions?

Open-source code development has become widespread in recent years. As a result, open-source software platforms have also become popular, and millions of developers from diverse locations are able to contribute to the same projects. On these platforms, various knowledge about them is obtained from user activity. This information is used in the form of developer metrics to solve a variety of challenges. In this study, we proposed new developer metrics, including commenting and issue-related activity, that require less information. We concluded that commenting on any feature of a project can be equally as valuable as code contribution. In addition, besides the quantitative ones, metrics based on only the existence of the activity have been shown to offer also considerable results. We saw that issues were crucial in identifying user contributions. Even if a developer makes a contribution to only one issue on a project, the relation between the developer and the project is tight. The hit scores are relatively lower because of the sparsity problem of our dataset; even so, we believe that we have presented improvable and remarkable new developer metrics.


Introduction
Thanks to the increasing capabilities of open source software development tools, the number of open source users and projects is growing each year. These platforms include millions of developers, each of whom has a different character and skill set, as well as a wide variety of projects that offer solutions to different problems. In environments with such a large amount of data, it is also difficult for developers to find similar products to their own, discern projects of interest, and reach projects to which they can contribute. As developers primarily use search engines or in-platform search menus to find projects, the constraints of text-based search [14] and challenges related to finding the correct keywords also cause some projects to be missed [10]. While various project recommendation systems are being developed to overcome this problem, projects must be rated by users for recommendation models to work properly. In the same way that viewers give ratings to movies that they have watched, developers need to rate the projects in which they are interested. However, this is not currently the case on (open-source) software development platforms. Several software and developer metrics are used to calculate the score that a user gives projects, which are extracted from the activity or features of both developers and projects.
Developer metrics which are used in many challenges include the number of lines of code, developers degrees of connection to one another, past experience, or common features (nationality, location, occupation, gender, previously used programming languages, etc.). These metrics offer solutions to different problems within open source software development and distributed coding, including automatic assignments (task, issue, bug, or reviewer) [5,1,11], project recommendation systems [28,22], software defect detection [18], etc.
In this study, new developer metrics are presented to be used for different problems. We developed a project recommendation system for the evaluation of metrics and obtained remarkable results. The recommendation system was developed based on a dataset consisting of data from GitHub. Most GitHub users are familiar with (i.e., contribute to or interest) relatively few projects hosted on the platform. Because of this, a critical sparsity problem has occurred. To address the problem, we selected a dataset with a high usersprojects ratio.
The structure of this paper is as follows. In the background section, we describe the literature on previously proposed metrics and project recommendation models for open-source software development platforms. In the research design section, the dataset used in this study is introduced and the proposed metrics are detailed. In the final section, the proposed metrics are assessed in terms of their accuracy rates in the project recommendation system.

Background
Pull request (PR) allows users to inform others about changes they have pushed to a branch in a repository on GitHub. PRs are a key feature in contributing code by different developers to a single project [7]. The proposed metrics related to this feature are used to solve different PR problems. PRs need to be reviewed by a reviewer to merge projects. If the result of a revision is positive, the PR is integrated into the master branch. Finding the correct reviewer is an important parameter for ensuring rapid and fair PR revisions. In this context, different metrics have been used to address the problem of automatic PR reviewer assignment. Existing literature has proposed various metrics to solve this problem, such as PR acceptance rate within a project, active developers on a project [11], PR file location [23], pull requesters social attributes [25], and textual features of the PR [27], among others.
Closing a PR with an issue, PR age, and mentioning (@) a user in the PR comments have all been used to determine the priority of a PR [26]. Cosentino proposed three developer metrics (community composition, acceptance rates and become a collaborator) to investigate project openness and stated that project owners could evaluate the attractiveness of their projects using these metrics [4].
Developer metrics are also used in the detection of software defects. In one study, defects were estimated using different metrics grouped by file and commit level. The number of files belonging to a commit, the most modified file of all files in a commit, the time between the first and last commits to a file, and the experience of a given developer on a committed file were identified as important metrics [18].
Reliability metrics are used to quantitatively express the reliability of a software product [12]. To measure reliability in open source projects, metrics such as number of contributors, number of commits, number of rows in commits, and certain metrics derived from them are used. Tiwari et al. proposed two important metrics for reliability: contributors and number of commits per 1000 code lines [24].
Code ownership metrics are also important for open source software. One study used the modified (touched) number of files to rank developer ownership according to code contributions [2]. In another study, the number of changed lines in a file (churn) was used to address this problem [15]. Foucault confirmed the relationship among these code ownership metrics and software quality [6].
Recommender systems are an important research topic in software engineering [9,19]. In ordinary recommendation models, previously known useritem matrices are used; in other words, the rating given by a user for an item is known. In this state, the essential research topic is on estimating with different algorithms and models the rating that the user has already given [20]. However, the point in question is different on open source software platforms. Considering the developer as the user and the project (repository) as the item, the rating given by a developer to a project is unknown. In this context, the first problem that must be solved is how to create an accurate developer-project matrix. At this point, different developer metrics come into play.
GitHub is the worlds largest code server, hosting more than 40 million repositories to which over 100 million developers have contributed 1 . As such, GitHub is a reasonable choice for developing a project recommendation model. Sun et al. relied on basic user activity to develop a model using GitHub data. Specifically, when rating a project for a developer, they used like-star-create activities related to projects [22]. We also used this scoring method to compare with our metrics in this paper. In another study, a developers social connections, programming with a common language, and contributions to the same projects or files were used as metrics [3]. Liu designed a neural networkbased recommendation system which used metrics such as working at the same company, previous collaboration with the project owner, and different time-related features of a project [13]. In a study aiming to predict whether a user would join a project in the future, the metrics used included a developers GitHub age (i.e., when their account was opened), the number of projects that they had joined, the programming languages of their commits, how many times a project was starred, the number of developers that joined a project, and the number of commits to a project [16].

Dataset
One of the most serious challenges in developing a recommender system is sparsity [17], a problem that occurs when most users rate only a few items [8]. This issue is present on GitHub because it is not possible for developers to be aware of the millions of repositories on the platform. In the studies mentioned in the previous section, we observed that limited (less sparse) data were used, which is contrary to the nature of the GitHub environment. Although the results obtained in these studies appear successful, the question remains how successful the proposed algorithms will be on real platform data. In light of this, a sub-dataset reflective of the sparsity problem inherent to GitHub was used in this study. The dataset contained all data related to 100 developers and 41,280 projects [29]. The creators of the dataset indicated that they selected the most active users on the platform. They then extracted all related data for these users from GitHub (commits, issues, pull requests, comments about these activities, watchers, etc.). Thus, we anticipated that the recommender system we developed would produce results parallel to those for the larger dataset of the platform as a whole. In this regard, although our evaluation results seem weak compared to similar studies, we believe that the proposed metrics are worthy of consideration.

Project Recommendation System
Designing a project recommender system for open source software development platforms includes two stages. First, the project-developer rating matrix is generated using specific metrics. Second, the top-k projects are recommended to each developer. Finally, the accuracy of the suggestions is evaluated. In this study, the recommendation model was designed as follows: 1. Different developer metrics were used to obtain the score matrix. The values of the features (metrics) were scaled from 0 to 10. As in the movie-user model, each developer will thus have given a rating (010) to each project. 2. The similarity between unknown projects 2 and rated projects was used to calculate the rating of unknown projects [22]. The similarity value between the projects was calculated using cosine similarity. When calculating the rating of an unknown one, the dot product of the similarity values between the projects that the user rated and the unknown project was used (Equation 1). An example scenario involving this calculation is presented in Figure 1.
3. The top 5 highest-rated projects among the unknown projects were recommended to each developer. 4. The accuracy of the recommendations was evaluated. Calculating unrated project with the help of similarity rated projects. 2 We assumed that unknown projects were those to which developers had no relationship and had made no contributions

Evaluation Techniques
While recommending projects to developers, there should be a ground truth for evaluating the proposed projects. Unlike ordinary recommender systems, there is an unsupervised model. The evaluation criteria for some studies related to this subject are set forth below.
-A user-project-rating matrix was split randomly into test and training subsets. Accuracy or recall scores were then calculated from the intersection between the top-n scores of the test and training subsets [22]. However, another study stated that this method should not be used on platforms like GitHub where time is an important parameter, pointing out that the problem of predicting past activity with future data will occur when using k-fold cross-validation by dividing the data randomly [11]. -In another study, the recommended projects accuracy was evaluated using the developers past commits to the related project. A recommendation was assumed to be correct if the number of commits belonging to a certain developer on the project was over a certain value. The average number of commits per project was set as the threshold value in the dataset [13]. -In a study predicting whether a developer would join a project in the future, the dataset was split into two different sets by time. In this way, the predicted result was verified with actual future data [16].
In this study, GitHubs watching feature was used as the ground truth. GitHub users can follow, or watch, projects whose developments they want to monitor [21]. If a developer is watching a project, this indicates that he/she is interested in the project. Thus, watching can be considered as a real evaluation criterion. In our model, the top-n projects were recommended to each developer. If the recommended projects were among the developers watched projects, the projects were considered a hit 3 . The case of a developer watching fewer than n projects was taken into account in the score equation 2.
The full name of a GitHub repository (project) is created by concatenating the owners username with the repository name. In analyzing our results, we noticed that the model recommended a project to a developer that only hit the owners namethat is, the model found an incorrect project by the correct owner. We evaluated this proposal has half the correct score, as recommending the correct owner to a developer will allow the developer to access the owners other projects.
An example scenario demonstrating this situation is given in Table 1. The projects recommended for Alice are listed in the first column. Since four of them are among the projects that Alice watches, the initial score is 4. In addition, there are two projects by a developer named "fengmk2 " among Alices watched projects (fengmk2/parameter and fengmk2/cnpmjs.org). For the fourth proposed project fengmk2/emoji, the owners name was guessed correctly, but the repository name was missed. In this case, Alice will be aware of other projects by "fengmk2". Thus a half-point is added to the initial score and 4.5 is the final score. In this way, a project recommendation model has been created for open source platforms. The algorithm of the recommendation model is presented in Figure 2, starting with selecting a feature as a metric and end with calculating hit scores.

Empirical Results
In this section, different developer metrics are given. These metrics provide information about a developers past activity on a project. All metrics used were scaled from 0 to 10 using the min-max normalization technique. The developerproject relationship was thus rated in the range 010 (as with a viewers rating of a movie). The results were calculated for the top 5 recommendation hit scores. We experimented with several metrics, including those based on coding language; obtained from the ratio of how many times a developer performed an activity to the total number of the activity; and created using different normalization methods. However, only metrics that obtained hit scores greater than 5% are shown in this study (except single metrics).

Single Metrics
Developer activity on projects was handled as a metric. Activity includes all kinds of comments, code contributions, revisions, and so on. In this section, all metrics were used individually in order to evaluate the significance of each. These metrics refer to the number of activities per project for a given developer ( Table 2).
All single metrics were given to the model individually, and the scores presented in Table 3  above. In addition to our metrics, another metric that extracted from the study of Sun et al. was added to make a comparison. They scored developers and projects using like-star-create activities. They used text data extracted from projects README and source code files to find project similarities [22]. There were approximately 22,000 repositories and 1,700 developers in their dataset, which was created using data from four groups of projects. We planned to use this less sparsed dataset to make a fair comparison but could not because the dataset was unshared. Owing to we could not communicate with them, we applied their rating algorithms to our dataset. When the results are analyzed, it is clear that the issue-related metrics are crucial even by themselves. Closing an issue with a PR means that the PR fixed a bug or issue in the project [26]. As our results indicate, issue closedwithPR is a remarkable metric. Opening an issue or PR is also an important metric. The most interesting conclusion that can be drawn from these results is that comments have higher hit scores than direct code contributions.

Fusion Metrics
In these results, we observed that some metric groups came to the forefront. New metrics can be proposed by grouping comments, code contributions, or other common featured metrics. In this context, fusion metrics were created from single metrics. The results of the fusion metrics are presented in Table 4.a. Most fusion metrics had a positive impact on hit score. comment in particular was a remarkable metric which showed that commenting is as important as code contributions. In addition, the issue related also drew our attention. The results of the ratio-based metrics were not as successful as the others. x pr2comment x commit2comment x code2comment x

Binary Metrics
We tried different metrics for the project recommendation in the previous section. It is interesting that the metric consisting only of comments achieved a higher success rate than the metric created by collecting all single metrics. We continued to study different metrics to analyze whether success could be increased with less information.
In the above single metrics, there is information about how many activities were made (For instance, John opened 18 issues in the projectX, the rating of John-projectX is 18). Alternatively, a set of metrics was created simply showing whether that activity existed (For instance, even so, John opened an issue on the projectX, the rating of John-projectX is 1 or there is no opened issue by John in the projectY, the rating of John-projectY is 0). In this context, the binary metrics were created using the equation 3.

Binary Fusion Metrics
Binary metrics and ratio-based fusion metrics consist only of 0s and 1s. As such, using them directly will not generate logical results. Therefore, binary fusion metrics were generated from these metrics, and only the results of these binary fusion metrics are given (Table 4.b). The results improved, and the top success metrics rankings swapped places. The best five metrics were the same as in the previous section. These results show that the number of comments is crucial in using comments as a metric. So, the presence of comments alone is not sufficient to use it properly. On the other hand, the issue related metric is leading among the binary metrics, indicating that issues are the most important feature for the developer-project relationship. Apart from our findings regarding the importance of comments, obtaining better results from issues than commits was one of the most surprising results of this study.

Conclusion
The main purpose of this study was to propose new developer metrics that could be used to solve different software engineering challenges. In this study, the features extracted from user activity on an open source platform (GitHub) were used. The study focused on finding metrics that would enable greater success with less knowledge. Some fusion metrics gave successfull results.
It is clearly seen that the comment made gave impressive results. For this metric, quantity is an important parameter. It means the more a developer writes comments, the more related to the project. On the contrary, it can deduce from the results of binary fusion metrics that the issue related is a quantity-free metric. It also means, to use this metric is adequate to know it's presence is whether or not. On this regard, It is revealing that issue is a significant feature for open-source platforms. In addition, metrics that used features based on existence (binary metrics) were highly successful, showing that for some activities, there is no need for quantities in order to extract knowledge.
We presented these new developer metrics, but we are curious why some of them became prominent. In light of this, we are planning another study involving a survey for junior and senior developers whom we can contact to understand the ground truth of our metrics success (especially the comment metric).
Because of the sparsity problem, our hit scores may not higher enough when comparing the other similar studies. Even so, we think that we offered some new improvable developer metrics. Moreover, to compare our results, we added a very similar metric used in Sun et al.s study [22], which revealed that most of our metrics surpassed that metric. In this context, we plan to apply the obtained metrics to different datasets for validity.
In addition, we anticipate that these metrics will be useful for solving various problems. Many developers besides owners and collaborators have contributed to projects due to the open source nature of GitHub. On some projects, external developers even made more contributions than the core team. These metrics can reveal developers contribution rankings on a project. To implement this, we plan to cooperate with a software company.