{'fraction_non_alphanumeric': 0.03304500608190296, 'fraction_numerical': 0.011200837951074469, 'mean_word_length': 5.155678036605657, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward the Automatic Classification of Self-Affirmed Refactoring

The concept of Self-Affirmed Refactoring (SAR) was introduced to explore how developers document their refactoring activities in commit messages, i.e., developers' explicit documentation of refactoring operations intentionally introduced during a code change. In our previous study, we have manually identified refactoring patterns and defined three main common quality improvement categories, including internal quality attributes, external quality attributes, and code smells, by only considering refactoring-related commits. However, this approach heavily depends on the manual inspection of commit messages. In this paper, we propose a two-step approach to first identify whether a commit describes developer-related refactoring events, then to classify it according to the refactoring common quality improvement categories. Specifically, we combine the N-Gram TF-IDF feature selection with binary and multiclass classifiers to build a new model to automate the classification of refactorings based on their quality improvement categories. We challenge our model using a total of 2,867 commit messages extracted from well-engineered open-source Java projects. Our findings show that (1) our model is able to accurately classify SAR commits, outperforming the pattern-based and random classifier approaches, and allowing the discovery of 40 more relevant SAR patterns, and (2) our model reaches an F-measure of up to 90% even with a relatively small training dataset.


Introduction
The role of refactoring has been growing from simply improving the internal structure of the code without altering its external behavior [1] to hold a key driver of the agile methodologies and become one of the main practices to reduce technical debt [2]. According to recent surveys, the research on refactoring has been focused on automating it through recommending candidate code elements to be refactored and which refactoring operations to apply [3,4,5,6]. Yet, more recent studies have shown that fully automated techniques are underused in practice [7]. Indeed, there is a need to minimize the disturbance of the existing design, by performing large refactorings, as developers typically want to recognize and preserve the semantics of their own design, even at the expense of not significantly improving it [7,8,9].
Therefore, several studies have taken a developer-centric approach by detecting how developers do refactor their code [10,11,12] and how they document their refactoring strategies [13,14]. The detection of refactoring operations and their documentation allows a better understanding of code evolution, and challenges that trigger refactoring, including the reduction of code proneness to errors, facilitation of API and type migrations, etc. While automating the detection of refactoring operations that are applied in the source code has advanced recently reaching a high accuracy [12], there is a critical need for a deeper analysis of how such refactoring activities are being documented. In this context, recent studies [13,14] have introduced a taxonomy on how developers actually document their refactoring strategies in commit messages. Such documentation is known as Self-Admitted or Self-Affirmed refactoring. Documenting refactoring, similarly to any type of code change documentation, is useful to decipher the rationale behind any applied change, and it can help future developers in various engineering tasks, such as program comprehension, design reverse-engineering, and debugging. However, the detection of such refactoring documentation was hardly manual and limited. There is a need for automating the detection of such documentation activities, with an acceptable level of accuracy. Indeed, the automated detection of refactoring documentation may support various applications and provide actionable insights to software practitioners and researchers, including empirical studies around the developer's perception of refactoring.
This can question whether developers do care about structural metrics and code smells when refactoring their code, or if there are other factors that may influence such non-functional changes. Furthermore, our previous study [14] found that there are several intentions behind the application of refactoring, which can be classified as improving internal structural metrics (e.g., cohesion, encapsulation), removing code smells (e.g., God classes, dead code), or optimizing external quality attributes (e.g., testability, readability). Yet, there is no systematic way to classify such refactoring related messages and estimate the distribution of refactoring effort among these categories.
To cope with the above-mentioned limitations, this paper aims to automate the detection and classification of refactoring documentation in commit messages. In particular, our objective is to analyze the feasibility and performance of applying learning techniques to (1) identify and (2) classify refactoring documentation based on commit messages. However, the detection of refactoring documentation is challenging, besides the inherited ambiguity of distinguishing meanings, in any natural language text, a recent study has shown that developers do misuse the term refactoring in their documentations [15], which hardens the reliance on that keyword alone. To cope with these challenges, we design our study to harvest a potential taxonomy that can be used to document refactoring activities. Such taxonomy is typically threatened by the potential false-positiveness of the collected samples. Therefore, we develop a baseline of code changes that are known to contain refactoring activities, and we analyze their commit messages, in order to ensure that the collected textual patterns are meant to describe refactoring, and so, to reduce false positives. Our study makes the following contributions: • We present a two-step approach that firstly distinguishes whether a commit message potentially contains an explicit description of a refactoring effort. Then, secondly classifies it into one of the three common categories identified in previous studies [13,14]. To the best of our knowledge this is the first attempt to automate the detection and classification of self affirmed refactorings.
• We evaluate the performance of our approach by comparing it against a keyword-based approach that relies on matching messages with known refactoring keywords [7,13,16,17,18,19]. Our key findings show that our model not only outperforms the keyword-based approach, but also accurately identifies refactoring related commits with an average accuracy of 98% and F-measure of 98%. Furthermore, we infer which features, i.e., keywords, are relevant for the detection of such refactoring documentation, and we extract them to extend the list of refactoring documentation patterns, identified in previous studies [13,14].
• We deploy our model as a lightweight web-service that is publicly available for software engineers and practitioners. We also publicly provide our dataset that served us as the ground-truth, for replication and extension purposes [20]. This paper is structured as follows. We start by explaining the notion of refactoring documentation (self-affirmed refactoring) and reviewing existing studies that are most related to commit messages classification in Section 3. Next, in Section 4, we detail our two-step classification methodology. More precisely, we elaborate on the data collection and preprocessing, choice of the classification algorithms. Then, we evaluate our approach, in Section 5, and report a comparative study between various classifiers, extracted from previous studies, and we identify most influential features. We report in Section 7 the threats to our work's validity, before concluding and describing our future work in Section 8.

Definition
Commit messages are the atomic descriptions of given code change, in natural language. It augments the change with human and machine readable meaning. In this study, we are interested in locating and automatically detecting developer's documentation of refactoring activities in commit messages. refactoring documentation is the textual description of what developers considers to be a refactoring performed in their code change. The act of intentionally documenting a refactoring activity is known as Self-Affirmed Refactoring (SAR) [14].
SAR is composed of a terminology that was found to be consistently used in refactoring-related commit messages. For example, if we consider the following commit message: Refactor createOrUpdate method in MongoChannelStore to extract methods and make code more readable The developer explicitly mentions the intention of refactoring, using the keyword "refactor ", along with providing extra information related to the refactoring activity: the developer reports 1) the type of refactoring operation performed, i.e., extract method ; 2) the code elements involved in the refactoring operation, i.e., createOrUpdate and MongoChannelStore; and 3) the intent behind the refactoring, i.e., make the code more readable. This message is labeled as Self-Affirmed Refactoring (SAR) as it totally or partially documents the refactoring performed in the source code.
The manual inspection of the message's corresponding commit 1 , reveals 3 methods extracted from the method createOrUpdate() that belongs to the class MongoChannelStore along with renaming a parameter to be consistent with the update. So, the documentation has given enough background to explain the rationale behind the refactoring (improving code readability), the operations performed and the code elements involved.
The complete list of identified SAR patterns, per category, is depicted in Finally, according to a recently published survey [45], refactoring is typically driven by intents that belong to the categories that we have used in this study.

Benefits
Commit messages are essential for not only comprehending code changes, but for many other aspects of software development, such as as classification of maintenance effort [40,42], code change summarization [46], files changeproneness and bug-proneness [47], etc. For instance, recent studies have shown the feasibility of extracting insights of software quality from developers inline documentation. For instance, mining developers comments has unveiled how developers knowingly commit code that is either incomplete, temporary, or faulty.
Such phenomenon is known as Self-Admitted Technical Debt (SATD) [48]. Similarly, our previous study has introduced Self-Affirmed Refactoring (SAR) [14], defined as developers explicit documentation of refactoring operations intentionally introduced as code change. Per analogy to SATD, SAR manifests as a positive phenomenon, known to be one of the primary concepts to manage technical debt [49]. So, it is of particular interest to understand how the developer's intent to refactoring code leads to an adequate corrective action, i.e., SATD resolution, especially that recent studies focus on understanding how SATD is being removed [50,51,52,53].
When it comes to refactoring documentation, revealing the intents that are frequently pushing developers to refactor, is of a major importance for the community, especially that recent surveys have shown that refactoring tools are under-used, and developers are still manually refactoring their code [7,54]. And so, these patterns can narrow the scope of refactoring towards what developers consider to be relevant, in order to bridge the gap between refactoring tools and their adoption in practice. However, the identification of these SAR patterns, is human-intensive, subjective, and error-prone. Coping with the burden of manual analysis is the main goal of this paper, by initially detecting then classifying these SAR patterns, into the above-mentioned categories. Furthermore, the automated identification of these SAR patterns, is not straightforward, as these keywords, are not necessarily exclusive to refactoring. Even refactoring, being the most intuitive keyword used to describe this activity, has been also found to be used out of its context [15].
Refactoring, just like any code change, has to be reviewed, before being merged into the code base. However, little is known about how developers perceive and review refactoring during the code review process, especially that refactoring, by definition, is not intended to alter to the system's behavior, but to improve its structure, so its review may differ from other code changes. Yet, there is not much research investigating the proper documentation of refactor-ing, which can facilitate the process of reviewing it. Through the identification of these SAR patterns, many example of documented refactorings can be provided for future investigations and analysis.

Related Work
In this section, we report studies related to developer's perception of refactoring and its documentation, along with the current state-of-the-art studies related to commit messages classification.

Refactoring and its documentation
A number of studies have focused recently on the identification and detection of refactoring activities during the software life-cycle. One of the common approaches to identify refactoring activities is to analyze the commit messages in versioned repositories. Stroggylos & Spinellis [17] searched words stemming from the verb "refactor" such as "refactoring" or "refactored" to identify refactoringrelated commits. Ratzinger et al. [16,18] also used a similar keyword-based approach to detect refactoring activity between a pair of program versions to identify whether a transformation contains refactoring. The authors identified refactorings based on a set of keywords detected in commit messages, and focusing on the following 13 terms in their search approach: refactor, restruct, clean, not used, unused, reformat, import, remove, replace, split, reorg, rename, and move. Peruma et al. [56] investigated how method, class, and package identifier names evolve and how this evolution was documented in the commit corresponding messages. By also analyzing their surrounding refactoring operations, the authors observed how names are influenced by their neighboring changes.
They extended their work by considering the situation where a rename is applied to an identifier whose data type is changed [57]. Such case is of an interest to the authors as it indicates a potential change in the behavior its associated identifier.
Prior work [58,13,14] has explored how developers document their refactoring activities in commit messages; this activity is called Self-Admitted Refactoring or Self-Affirmed Refactoring (SAR). In particular, SAR indicates developers' explicit documentation of refactoring operations intentionally introduced during a code change.
The existence of such patterns unlocks more studies that question the developer's perception of quality attributes (e.g., coupling, complexity), typically used in recommending refactoring. For instance, AlOmar et al. [59] identified which quality models are more in-line with the developer's vision of quality optimization when they explicitly mention in the commit messages that they refactor to improve these quality attributes. This study shows that, although there is a variety of structural metrics can represent internal quality attributes, not all of them can measure what developers consider to be an improvement in their source code. Based on their empirical investigation, for metrics that are associated with quality attributes, there are different degrees of improvement and degradation of software quality.
Hattori and Lanza [62] developed a lightweight method to manually classify history logs based on the first keyword retrieved to match four major development and maintenance activities: Forward engineering, Re-engineering, Corrective engineering, and Management activities. Also, Mauczka et al. [67] have addressed the multi-category changes manually using three classification schemes from existing literature. Tsantalis et al. [35] conducted a multidimensional empirical study on refactorings and performed a systematic labeling of the commit messages to better understand the purpose of the applied refactorings. Silva et al. [10] applied a thematic analysis process to reveal the actual motivation behind refactoring instances after collecting all developers' responses. Further, a few studies [28,69] propose the classification of refactoring instances as root-canal or floss refactoring through the use of manual inspection.
Yan et al. [68] used discriminative topic modeling techniques to automatically classifying software changes.
Mockus & Votta [60] designed an automatic classification algorithm to classify maintenance activities based on a textual description of changes. Another  [64] proposed an automated technique to classify commits into maintenance categories using seven machine learning techniques.
To define their classification schema, they extended Swanson's categorization [73] with two additional changes: Feature Addition, and Non-Functional. They observed that no single classifier is the best. Another experiment that classifies history logs was conducted by Hindle et al. [41], in which their classification of commits involves the non-functional requirements (NFRs) a commit addresses.
Since the commit may possibly be assigned to multiple NFRs, they used three different learners for this purpose along with using several single-class machine learners. Amor  In their extended work [70], the RF model showed a promising accuracy of 76%.
More recently, a replicated study [71] of [43] introduced code density of a com-mit to study the purpose of a change. Using code-density based classification, they achieved up to 89% accuracy for cross project commit classification using LogitBoost classifier.
In this paper, we build on top of these techniques to leverage an automated identification and classification of SARs. Although the manual summarization of SAR is useful, it is considered as a time-consuming task because of the required manual effort to derive the list of patterns. Although much work has been done on automatically classifying commits in general, there is no currently automatic way to identify SAR patterns specifically. Several studies [40,43,61,62,64,66,67,68,70,71] have discussed how to automatically classify change messages into Swanson's general maintenance categories (i.e., Corrective, Adaptive, Perfective). Refactoring, in general, has been classified as a sub-type of "Perfective" in this maintenance category. Currently, there is no study that reports specific subcategories of refactoring extracted from realworld scenarios of commit messages and performs an automated classification of SAR commits. Therefore, in this paper, we push the Self-Affirmed Refactoring research a step forward by introducing an automatic classification approach to (1) determine whether a commit contains SAR or not (cf., Table 1), and (2) classify SAR into its three categories (see Table 2). Compared with the patternbased approach, our automated approach can identify more SAR patterns that can complement and extend the list of patterns identified in [14].
Further, in this work, we are detecting the indicators of refactoring to understand how developers document refactoring. We are not labelling refactoring operations themselves; we are instead labelling the commit messages that are found to contain refactoring operations. The existence of refactoring operations, in the studied commits, can be verified by running state-of-the-art tools, such as Refactoring Miner [12] and RefDiff [11] tools. Both of these studies indicated that their tool achieves high accuracy (precision of 98% and 100%, and recall of 87% and 88%, respectively), which gives us confidence to use one of these tools as a form of validation that the commits contain refactoring.
As can be seen in table 3, commit messages are extensively used in existing literature to classify several maintenance-related tasks. Studies that focus on classifying bug and feature requests have used commit messages as a primary source of information to generate high accuracy and applicable results. However, our approach is not restricted to a specific source of textual information. Future work could replicate our approach with other types of metadata, e.g., issue descriptions.

Approach
In this section, we first provide an overview of our approach. Then, we elaborate on the technical details of the adopted classification technique, in the following subsection. The overview of our approach is depicted in Figure 2, and a sample of commit messages is demonstrated in Figure 3.

Data Collection
To collect the necessary commits, we refer to an existing large dataset of links to GitHub repositories [74]. We perform an initial filtering, using Reaper [75], to only navigate through well-engineered projects while verifying that they were Java-based; the only language supported by Refactoring Miner. The authors of this dataset classified "well-engineered software projects" based on the projects' use of software engineering practices such as documentation, testing, and project management. So, we ended up reducing the number of selected projects from 57,447 to 3,795.
Using "well-engineered" and "well-documented" kind of interchangeablyalthough we acknowledge the potential value of having a more diverse set of projects, and our findings may not extend to projects that are not as welldocumented, because our primary research methods rely on documentation, we chose to focus on projects that would be likely to have high-quality documentation (i.e., commit messages) consistently available.

Refactoring Detection
To extract the entire refactoring history in each project, we use the popular refactoring mining tool, i.e., Refactoring Miner [10]. Our choice to use Refactoring Miner is justified by the fact that it achieved the highest accuracy in detecting refactorings compared to the state-of-the-art available tools, with a precision of 98% and recall of 87% [10,12] along with being suitable for our study that requires a high degree of automation in data mining. In this phase, We collect a total of 1,208,970 refactoring operations from 322,479 commits, applied during a period of 23 years (1997-2019).

Overall Framework
In a nutshell, the goal of our work is to automatically identify then classify commit messages containing refactoring documentation, i.e., Self-Affirmed Refactorings, (for the sake of simplicity, we refer to them as SAR). Our approach takes as input, a commit message, and makes a binary decision on whether it contains SAR or not. If a SAR is detected, it classifies it into one of of three common categories: (i ) internal quality attribute (ii ) external quality attribute, and (iii ) code smell [14]. The overall framework of our approach is depicted in Figure 2. We formulate a two-phased approach that consists of a model building phase and a prediction phase. In the model building phase, our goal is to build a model from a corpus real world documented refactoring operations (i.e., commit messages). In the prediction phase, the model created in the previous phase will be used to predict categories of new refactoring-related commit messages.
Our framework takes commit messages along with their ground truth categories obtained by manual inspection as input for the training procedure extracted from different projects, provided by a previous study [14]. Based on this input, the commit messages are preprocessed, allowing for informative featurization. Next, for each commit message, we extract features (i.e., words) to create a structured feature space. Then, we use the extracted features to build the training set. In total, we experimented 9 commonly used classifiers to evaluate our model for prediction. We selected these classifiers as they are typically used in previous commit classification studies as well as several software engineering classification/prediction problems [41,43,61,64,65,70,71], as outlined in Table   3. After training all models, we use a testing set to challenge the performance.
Since the model has already learned the vocabulary of N-Gram (discussed in Section 4.4.3) and their weights from the training dataset, we extract features from the test data based on that vocabulary and weights, and input them to the model. Finally, the classifier will output the predicted label for each tested commit message.

Commit Classification
Our classification process has five main phases: (1) data preparation, (2) text cleaning and preprocessing, (3) feature extraction using N-Gram, (4) model training and building, and (5) classifier selection and model evaluation. Since a commit message is written in plain text, we follow the approach provided by Kowsari et al. [76] that discussed a recent trend in text classification techniques and algorithms.

Data Preparation
Our goal is to provide the classifier with sufficient commits that represent the categories analyzed in this study. Since the number of candidate commits to classify is large, we cannot manually process them all, and so we need to randomly sample a subset while making sure it equitably represents the featured classes, i.e., categories. Since an imbalanced training dataset or class starvation (i.e., not having adequate instances of a certain class) could worsen the performance of the model [43,70], we make sure that the classes for two-class (i.e., with or without SAR) and multiclass (i.e., Internal QA, External QA, and Code Smell) classification problems are equally distributed when preparing the data for the training (cf. Table 4). The classification process has been performed by the authors of the paper. To approximate the needed number of commits to add, we reviewed the thresholds used in the studies related to commit classification (see Table 3). The highest number of commits used in comparable studies was around 2,000 commits [40,64,69]. Thus, we select a sample of 2,000 commits from 3,795 projects for each classification model. Below we detail the manual analysis of the data we use for our classification. For data preparation, building the ground truth is challenging since we are looking for a particular set of commits. To prune the search space, we started with using an existing dataset of commits [14], manually inspected and validated for containing refactoring operations and an associated description at the commit message. We intend to build our own dataset by choosing a subset of this dataset, in a way to serve the purpose of the binary and multiclass classification.
To prepare the dataset of the binary classification, we need to create two groups of commits, i.e., commits with or without SAR. The first group (with SAR) is created by randomly sampling commits, previously known to contain SAR patterns listed in Table 1. We further perform another round of individual verification by the authors before adding them to the group. Commits for which there was no full agreement by the authors were excluded from our dataset. The second group (without SAR) can be easily created by randomly choosing commit messages that simply do not contain these SAR patterns, but since we do want to strengthen our decision boundary, we intend to choose commits that are closest to neighboring regions between the two classes. To do so, for each commit from the first group (with SAR), we locate the set of its contiguous commits (committed either before or after), and performed by the same committer, then we randomly sample one of them to be added to the second group (without SAR), after manually verifying that it does not contain any description of a refactoring activity.
For the multiclass classification, we build it by making sure the chosen commits belong to one of the three categories listed in Table 2. To avoid involving our interpretation, it is important to note that the description of the categories listed in Table 2 needs to be explicit in the commit messages. We used stratified sampling to select 2,000 commit messages for manual classification, divided equally for each stratum. To ensure that these commits reported developers' intention to perform refactoring, and to improve quality attributes or fix code smells, we inspected these commits to remove false positives.
To avoid having false positive commits, we applied the filtering to narrow down the commit messages eliminating the ones that are less likely to be classified as self-affirmed refactoring. We designed the filtering to help ensure that we only trained the algorithm on higher-quality commit messages [77].
We followed the process from existing papers in filtering commit messages [66,78,79]. For example, Fu et al. [78] filtered out short commit messages.
Mauczka et al. [66] used the "Blacklist" category to filter all commits, which underlying modifications were not carried out by humans or which do not actually include any source code modifications. In our work, we apply five filtering heuristics to narrow down the commit messages eliminating the ones that are less likely to be classified as SAR. It is important to note that we removed short commit messages from the training, but not from the testing set because (1) short commit messages do not contain enough information and do not clearly describe the purpose of code change , and (2) we want to train the classifier on well-documented commit messages, and label commits that contain enough information about refactorings. Prior study has pruned short commit messages since these will be noise for the classifiers, and they did not record the cause of the changes [78]. Some criteria we used for filtering were as follows: • If a commit contains an alternative form of the word "refactor" such as "re-factor*", the commit was classified as SAR commit.
• If a commit message contains a pattern that is in a slightly different form of one of the patterns, such as "simplify the code" and "simplify code", the commit was classified as SAR commit.
• Commits that were either too short or ambiguous were discarded. Some examples of hard-to-classify commit messages are: "Solr Indexer ready" 2 , "allow multiple collections" 3 , and "Auto configuration of AgiScripts" 4 .
• If one commit could belong to more than one class, it was excluded.
• If the quality attribute is a part of the identifier name, the commits were excluded, e.g., "SONARJS-541 Precise issue location for ExpressionComplexity (S1067)". We discarded this commit because "complexity" is referring to a part of a class name and not a quality attribute.
The above-mentioned examples of ambiguous commit messages prevent us from being confident, and hence, for each discarded commit message, we randomly sampled another replacement. We repeated this process until we found the commit message that we were able to confidently classify. Because of the random nature of the process, some classes were saturated faster than others, so we kept increasing the number of instances only for the underrepresented classes, until we find the right balance between all classes. The criteria listed above reduced the number of commits and helped us focus on the most insightful commit messages. For the binary classification, 177 commits were removed because of them either being short or ambiguous. Also, in our case, any message with less than 7 characters was too short for us to decide. The evaluation resulted in keeping 1,823 commits and 1,044 commits, respectively for two-class and multiclass classification problems. To mitigate the risk of having a biased dataset and to inspect the level of agreement of the manual classification, we extract stratified sample of our dataset that are classified by the first author, and have these sample commits independently classified again by the second author.
Particularly, similar to [70], in order to inspect manual classification agreement, we randomly classified a 10% sample of commits, i.e., 186 and 105 commits out of the 1823 and 1044 for two-class and multiclass classification problems, respectively. This quantity roughly equates to a sample size with a confidence level of 95% and a confidence interval of 8. We used Cohen's Kappa coefficient [80] to evaluate the inter-rater agreement level for the categorical classes. We achieved an agreement level of 0.96 for the two-class classification, and 0.87 for multiclass classification. According to Fleiss et al. [81], these agreement values are considered to have an almost perfect agreement (i.e., 0.81-1.00).
The result of this classification is available in the reproduceability package of this work, thus, it can be reused and extended [20].

Text Cleaning & Preprocessing
After the data preparation phase, we applied a similar methodology explained in [76,82] for text pre-processing. In order for the commit messages to be classified into correct categories, they need to be preprocessed and cleaned; put into a format that the classification algorithms will process. This way, the noise will be removed, allowing for informative featurization. To extract features (i.e., words), we preprocess the text as follows: • Tokenization: The goal of tokenization is to investigate the words in a sentence. The tokenization process breaks a stream of text into words, phrases, symbols, or other meaningful elements called tokens [76]. In our work, we tokenize each commit by splitting the text into its constituent set of words. We also split tokens on special characters (e.g., the string "package-level" would be separated into two tokens, "package" and "level").
• Lemmatization: The lemmatization process either replaces the suffix of a word with a different one or removes the suffix of a word to get the basic word form (lemma). In our work, the lemmatization process involves sentence separation, part-of-speech identification, and generating dictionary form. We split the commit messages into sentences, since input text could constitute a long chunk of text. The part-of-speech identification helps in filtering words used as features that aid in key-phrase extraction.
Lastly, since the word could have multiple dictionary forms, only the most probable form is generated.
• Stop-Word Removal: Stop words (i.e., words and common English words such as "is", "are", "if", etc) are removed since they do not play any role as features for the classifier [83].
• Capitalization Normalization: Since text could have a diversity of capitalization to form a sentence and this could be problematic when classifying large commits, all the words in the commit messages are converted to lower case and all verb contractions are expanded.
• Noise Removal: Special characters and numbers are removed since they can deteriorate the classification. More specifically, we remove all numeric characters, unique and duplicate special characters, email addresses and URLs.

Feature Extraction Using N-Gram
After cleaning and preprocessing the text, we apply feature extraction to extract only the most useful information from text strings to differentiate classes in both classification problems. In particular, we selected the N-Gram technique for feature extraction. The N-Gram technique is a set of n-word that occurs in a text set and could be used as a feature to represent that text [76].
In general, N-Gram term has more semantic than an isolated word. Some of the keywords (e.g., "improve") do not provide much information when used on its own. However, when collecting N-Gram from commit message (e.g., Refactor:Remove redundant method names, extract method, improve usability), the keyword "improve" clearly indicates that this is a SAR commit. In our classification, we use bigrams since it is very common to enhance the performance of text classification [84], and we select Fisher Score filter-based feature selection [85,86] to featurize text and manage the size of the text feature vector, similar to [82]. As for the weighting function, we used the standard Term Frequency-Inverse Document Frequency (TF-IDF) [87] due to its popularity in the research community (the value for each N-Gram is its TF score multiplied by its IDF score). Thus, each preprocessed word in the commit message is assigned a value which is the weight of the word computed using this weighting scheme. TF-IDF gives greater weight (e.g., value) to words which occur frequently in fewer documents rather than words which occur frequently in many documents.

Model Training and Building
In this phase, we performed the 10-fold cross-validation technique to assess the variability and reliability of the classifier. Specifically, for each of the classification methods, we combined the commit messages into a single large dataset. Then, we split the dataset into ten folds, where each fold contained an equal proportion of commit messages. Thereafter, we performed ten evaluation rounds with different testing dataset in which nine folds were used as training dataset and the remaining one of the ten folds is used as the testing dataset.
We aggregated the results of the ten evaluation rounds and reported the average performance for each classifier.

Classifier Selection and Model Evaluation
Selecting the proper classifier for optimal classification of the commits is a rather challenging task [88]. Best practices suggest that developers document their commits by providing a commit message along with every commit they make to the repository. These commit messages are usually written using natural language, and generally convey some information about the commit they represent. In this study, we are dealing with two-class and multiclass classification problems since the commit messages are categorized into two and three different types as explained in Table 1  ies [43,70,82]. These studies found that Random Forest (RF) achieves high performance. We investigated each classifier ourselves using common statistical measures (precision, recall, accuracy, and F-measure) of classification perform-ance to compare each. It is important to note that the calculation of F-measure for multiclass classification is not supported by Azure Machine Learning (Azure ML). Thus, to facilitate comparison and to have all statistical measures that are consistent with two-class classification, we compute F-measure for multiclass in terms of precision and recall using the following formula: where Precision (P) and Recall (R) are calculated as follows: It is worth noting that a few models that we consider are inherently binary classifiers. In order to adjust for multiclass classification, each classifier applies the One-vs-All strategy for issues that require multiple output classes [89]. Thus, to ensure fairness, we use One-vs-All strategy for multiclass classification when using the following five classifiers: Gradient Boosted Machine (GMB) [90], Support Vector Machine (SVM) [91], Locally Deep SVM (LD-SVM) [92], Averaged Perceptron Method (APM) [93], and Bayes Point Machine (BPM) [94]. The remaining classifiers, consider in this study, are: Logistic Regression (LR) [95], Random Forest (RF) [96], Decision Jungle (DJ) [97], and Neural Network (NN) [98]. Our experiment is conducted using Microsoft Azure Machine Learning (Azure ML [99]), as it provides a built-in web-service once the classification models are deployed.

Results & Discussion
In this section, we conduct an empirical study to assess the performance of our approach. To evaluate different commit classification models, we used standard statistical measures to measure the performance of the classification (Precision, Recall, Accuracy and F-measure). In the following, we report the results of our research questions.
Replication package. We provide our comprehensive experiments package available in [20] to further replicate and extend our study. 5.1. RQ1: Is it possible to accurately perform two-class and multiclass SAR classification using our machine learning technique?
As shown in previous work [14], SAR can be extracted from commit messages. However, there is a lack of automatic techniques to classify them. In this work, we performed an automated approach to classify SAR to determine if the classification using machine learning techniques can result in high accuracy. A comparison between classification algorithms is reported in Table 5 and 6. The best performing model was used to classify the test dataset. Based on our findings, the F-measure of Random Forest (RF) and Gradient Boosting Machine (GBM) are respectively 98% and 98% which are clearly higher than their competitors for the two-class classification. For the multiclass, in addition to RF and GBM, Logistic Regression (LR) outperforms the other models with F-measure of 93%. Figures 4, 5, and 6 show the detailed performance for the best multiclass classifiers.
Random Forest and boosting learning machines belong to the family of ensemble learning machines, and have typically yielded superior predictive performance mainly due to the fact that they both aggregate several learnings.
As for Logistic Regression, the fact that Logistic Regression achieves comparable performance as Random Forest and Boosting can be explained by the fact that the underlying true model for the text data has an inherent structure that matches the logistic regression assumption.
Another observation with regard to the classifiers accuracy is that few of the classifiers we considered in our study (GBM, SVM, LD-SVM, APM, and BPM) are inherently binary classifiers, and we used the One-vs-All strategy to adapt them for multiclass. Hence, these classifiers give us higher accuracy when performing binary classification compared to multiclass classification (98% vs 93%, respectively).Another reason for getting a different accuracy value when identifying multiclass labels vs two-class is that some commit messages could potentially belong to multiple categories. Hence, the machine learning classifiers, considered in this study, got confused when classifying such commit messages. Figures 7 and 8 show two cases of commit messages that confused the classifiers when performing two-class and multiclass classifications, respectively. The first commit message (Figure 7) contains a pattern (i.e., changing package name) that is a synonym of the patterns "renam*" or "use better name". The second commit message (Figure 8) contains more code element-related keywords such as "method" or "class" tend to be classified as "Code Smell" since code smellrelated commits usually contain more description of the code elements that need to be optimized. This commit example targets to improve the flexibility at the design phase, which should be classified as "External QA".
Moreover, it is important to note that the classes used in this study categorize mainly the refactoring documentation and do not reflect the overall activities of the commits. Hence, these commit messages do not strictly contain refactoring code changes, especially that we noticed that refactoring tends to be interleaved with other software engineering tasks, such as fixing bugs, migrating type changes etc. Therefore, it is important to consider such context to better understand the intention behind the application of refactoring, and this will be our main future research direction.
Summary. We find that our approach is accurately identifying the SAR patterns and the three common quality improvements with an F-measure of 98% and 93% for the two-class and multiclass classification problems, respectively.

RQ2
: How effective is our machine learning approach in classifying SAR?
The main goal of this research is to propose an automatic approach to classify SAR commits that can effectively outperform the classification over the current state-of-the-art baselines, i.e., Pattern-based [14] and Random classifier [79]. The selection of the two baseline approaches to compare against our approach was similar to Da Silva et al. [79]. We opt to choose a patternbased approach because the methods used so far to identify refactoring commits [7,15,16,17,19,31,54,66] and analyze refactoring activity [19] heavily    rely on string matching. Other studies (e.g., [66]) that focused on classifying commit messages on Swanson's categories (Corrective, Adaptive, Perfective) also used keyword-based approach. Currently, there is no evidence on how well pattern-based approaches perform. The choice of random classifier was similar to [79] that assumes that the detection of self-affirmed refactoring is random.
Existing studies (cf., Table 3) that have applied machine learning techniques in similar contexts (i.e., text classification) usually evaluate their approach using different classifiers. To compare their approach against others, they consider the keyword-based approach. To our knowledge, the only study that considers additional approach (i.e., random classifier) is the study by Da Silva et al. [79].
Thus, we consider keyword-based and random classifiers to compare against our approach.
Answering this research question would shed light on whether the classification of SAR is a learning problem or not. We hypothesize that if learning algorithms cannot outperform a String matching algorithm, then there is no need for proposing such a framework. The two chosen baselines, for this investigation, are listed below: • Baseline 1 (Pattern-based technique): The pattern-based approach in identifying SAR is proposed by AlOmar et al. [14]. In their work, they identified 87 recurring patterns in SAR commit messages. We use these patterns as indicators of refactoring activities, i.e., if a pattern exists in a commit, it is then classified as a SAR.
In order to calculate the standard statistical metrics for this baseline, we use a set of 1,823 and 1,044 commit messages (cf., Table 4 Thus, using the tp, tn, fp, and fn values, we compute the precision, recall, and F-measure. • Baseline 2 (Random classifier): Similar to Da Silva et al. [79], we consider Random classifier as one of the baselines to compare against our approach. The rationale behind using this random classifier to hold our approach accountable for providing significantly better results in comparison with a random classification. The precision of this approach is calculated by taking the total number of SAR over the total number of commit messages for all projects. As for the recall, there is a 50% chance that commit messages will be classified as SAR. The calculation of F-measure is explained previously in Section 4. Table 7 and Figure 9 present the experimental results of our approach com-pared with baseline 1 (Pattern-based), and baseline 2 (Random classifier). For our approach, we consider the highest F-measure score to compare against the other two baselines. Our approach provides an improvement over the comment patterns, outperforming it by 1.53 times and 1.45 times for two-class and multiclass respectively. We can see from Table 7 that our approach outperforms the simple Random baseline by 1.84 times and by 22.14 times respectively for two-class and multiclass classifications.
To better analyze our findings, after deploying our models as a web-service, we validate the two-class and multiclass models by randomly selecting 500 and 363 new commit messages, respectively. These new commit messages contain all types of commits (e.g., short commit messages, commits with more than two classes, and commits with quality attributes as part of the identifier names).
We manually read through commit messages that were classified as SAR commits in the prediction phase, and were classified as non SAR according to the pattern-based approach. Intuitively, such results induce the existence of features that represent the refactoring activity, and they are not captured by the previous study. Indeed, we found a set of featured keywords that do indicate refactoring activities (e.g., "Tidy code", "repackage", and "fix bad merge and coding style issues"), and were not reported by any of the previous studies related to refactoring documentation. Such featured patterns could complement the list of manually identified 87 SAR patterns. Figure 10 reveals examples of these new patterns.  After assessing the accuracy of our approach in classifying SAR commits, we want to investigate the amount of training data that is needed to effectively classify SAR. Our approach will be easily extended if a small dataset can be used for SAR identification. On the other hand, if a large number of commits are required, then our approach requires considerable time and effort.
To answer this research question, we incrementally add training data and assess the classifier's performance. We start by randomly selecting a stratified sample of 11,000 commits for each stratum (i.e., SAR and non-SAR commits) provided by the authors of [14], and combining these commits into a single large dataset. Then, we follow the classification process discussed in Section 4, which results in 5,000 equally divided for each class. We then split the dataset into 10 folds with equal size, ensuring that each partition has the same ratio of SAR and non-SAR commits. For the multiclass classification problem, however, we use only a stratified sample of 1,044 commits discussed in Section 4. The reason for only considering these commits is that we are restricted by the minimum number of commits belonging to the code smell category. Thus, to avoid having an imbalanced training dataset, we keep the training size as it was originally set up. We discard the 44 commits since this number is less than the selected batch training size. For both classification methods, we run our approach using the 10-fold crossvalidation technique, using nine folds as training data and the remaining one for testing. Because our target is to examine the impact of the quantity of training data on the performance of the classification, we train the classifier adding batches of 100 commits at a time similar to [79], and evaluate their performance on the testing dataset. For each batch of commits, we maintain the same ratio of SAR and non-SAR commits. The process ends when all of the training dataset is used. After each iteration, we report the average performance for all of the folds. Figure 11 reveals the F-measure scores when identifying SAR and non-SAR commits. Overall, we find that the F-measure maintains almost the same level with no significant improvement, in terms of accuracy, as the dataset size increases. As can be seen, we obtain a high F-measure value starting with less than 1000 commits. We conclude that only one fold of the training dataset is sufficient to identify SAR commits with F-measure of 90%. To achieve F-measure higher than 90%, at least one fold of 1000 is needed. Figure 12 shows the F-measure values when classifying Internal QA, External QA, and Code Smell commits (multiclass). In general, we notice that the F-measure value slightly increases as we increase the number of commits in the training dataset. To get at least 90% F-measure, more than 400 commits are needed. We conclude that to achieve a performance equivalent to 80% and 90% of the high F-measure score, only 10% and 40% of the commit messages are required respectively. To test the significance of the difference in F-measure values, we applied the Mann-Whitney U Test and found that the differences are not statistically significant.
Summary. We find that to achieve a performance equivalent to 90% of the high F-measure score, only one fold of commit messages is required for the two-class and multiclass classification problems, respectively.

Research Implications
This section further discusses positions our work in the spectrum of existing studies and how it implicates current research and practice. This work will help refactoring recommending systems by narrowing their scope (e.g., working on code fragments that developers are interested in). Current recommender system did not look at the intention, they excluded completely the intention of developers. Thus, these recommender systems are underused because they did not consider this important aspect.

Implications for researchers
From a research perspective, recent studies have been focusing on automatically identifying any execution of a refactoring operation in the source code [11,12,105]. The main purpose of the automatic detection of refactoring is to better understand how developers cope with their software decay by extracting any refactoring strategies that can be associated with removing code smells [3,4], or improving the design structural measurements [5,106]. However, these techniques only analyze the changes at the source code level, and provide the operations performed, without associating it with any textual description, which may infer the rationale behind the refactoring application. Our proposed model intends to bridge this gap by complementing the existing effort in accurately detecting refactorings, by augmenting with any description that was intended to describe the refactoring activity. As previously shown in Tables 1 and 2 [36,107,108]. One of the main limitations of these studies is the absence of any context related to the application of refactorings, i.e., it is not clear whether developers did apply these refactoring with the intention of improving design metrics. Therefore, the use of our model will allow the consideration of commits whose commit messages specifically express the refactoring for the purpose of optimizing structural metrics, such as coupling, and complexity, and so, many empirical studies can be revisited with a more adequate dataset. Therefore, our model can help to build a corpus of refactoring descriptions, and so many studies can better analyze the typical syntax used by developers in order to develop better natural language models to improve it, and potentially automate it, just like existing studies related to other types of code changes [109,110,111].
This work can help researchers to investigate the consistency between code changes and the actual intention and explore whether there is an overlap or not.

Implications for educators
From an educator point of view, this study helps to teach the new generation of developers or engineers the best practice to document their refactoring activity.

Threats to Validity
In this section, we identify potential threats to the validity of our approach and our experiments.
Construct Validity: Since our approach heavily depends on commit messages, we used well-commented Java projects when performing our study. Thus, the quality and the quantity of commit messages might have an impact on our findings. Additionally, a well-commented project might not contain SAR as developers might not document refactoring activities in the commit messages.
We mitigate this risk by choosing projects that are appropriate for our analysis.
Another potential threat relates to manual classification. Since the manual classification of training commit messages is a human intensive task and it is subject to personal bias, we mitigate manual classification related errors by discarding short and ambiguous commits from our dataset and replacing them with other commits. Another important limitation concerns the size of the dataset used for training and evaluation. The size of the used dataset was determined similarly to previous commit classification studies, but we are not certain that this number is optimal for our problem. It is better to use a systematic technique for choosing the size of the evaluation set. Concerning the relationship between refactoring and quality issues, we designed our study with the goal of classifying refactoring documentation. We have not explored if the refactoring operations detected by the Refactoring Miner tool are related to the corresponding quality issues documented by developers in the commit messages. Further, recent studies [68,112,113] indicate that commit messages could capture more than one type of classification (i.e., mixed maintenance activity). Figure 13 shows a commit message could belong to internal quality attribute (since it discusses code complexity reduction), external quality attribute (since it points out scalability improvement), and code smell (since it explains duplicate code removal). In this work, we have not yet investigated whether a significant number of commits can belong to more than once class, and if so, we plan on exploring a multi-label classification in our future work. Although we used commit messages as our primary source of text, our approach is not restricted to a specific source of textual information. In our future work, we can replicate our approach with other types of metadata, including issue descriptions. For this study, we chose to focus on commit messages rather than issue descriptions, since issue descriptions can be very high level, may not go into code change details, may not always be available, and may refer to multiple changes in the code that span or mix different purposes (e.g., bug fix and feature request). Besides, not all projects are using issue tracker. If the issue tracker is guaranteed to be available, it could be used as an additional source of information.
The use of well engineered projects is a double-edged sword, while it guarantees an easier labeling process, and providing less noisy data for the approach, it hinders its generalizability since these projects represent only a subset of all projects. So, our model may not achieve similar (high) performance across many projects. We tried to mitigate this concern by considering different types of projects, belonging to different domains. We shuffled commit messages during the training and testing to avoid any biases.

Conclusion
In this paper, we proposed an approach to identify and classify self-affirmed refactoring in commit messages. We compared the performance of our approach to pattern-based and simple random baselines. Our results show that our approach (1) is able to accurately classify SAR commits with accuracy of 98% and 93% for two-class and multiclass classification methods, respectively, outperforming the two state-of-the-art approaches considered in this study, and (2) can achieve F-measure of 90% using only 1% and 40% of the commits when performing two-class and multiclass SAR classifications respectively. This indicates a relatively small training dataset is sufficient to classify SAR commits.
In the future, we plan to study the applicability of our approach to other projects developed in different programming languages, and to other domains.
Another potential research direction is to use the current findings to build a tool that supports the identification and detection of self-affirmed refactoring commits. We also plan to conduct different user studies with our industrial partner predict the refactoring intention of the developers and further assess whether it aligns with what happened to his source code after applying refactoring.