{'fraction_non_alphanumeric': 0.02404783569478747, 'fraction_numerical': 0.01013908748212661, 'mean_word_length': 4.677859778597786, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Story Point Effort Estimation by Text Level Graph Neural Network

Estimating the software projects' efforts developed by agile methods is important for project managers or technical leads. It provides a summary as a first view of how many hours and developers are required to complete the tasks. There are research works on automatic predicting the software efforts, including Term Frequency Inverse Document Frequency (TFIDF) as the traditional approach for this problem. Graph Neural Network is a new approach that has been applied in Natural Language Processing for text classification. The advantages of Graph Neural Network are based on the ability to learn information via graph data structure, which has more representations such as the relationships between words compared to approaches of vectorizing sequence of words. In this paper, we show the potential and possible challenges of Graph Neural Network text classification in story point level estimation. By the experiments, we show that the GNN Text Level Classification can achieve as high accuracy as about 80 percent for story points level classification, which is comparable to the traditional approach. We also analyze the GNN approach and point out several current disadvantages that the GNN approach can improve for this problem or other problems in software engineering.


I. INTRODUCTION
With the importance of story point effort estimation, there are researches on how to apply the technique to automatically predict the story point of a software task/ issue. To our knowledge, one of the well-known datasets of software effort estimation is the dataset published by Choetkiertikul et al [1]. Machine learning-based approaches have been applied to this dataset for story point estimations by [1]. The approaches range from the classical approach from Bag of Word (BOW) [2] to modern deep learning approaches from Recurrent Highway Network (RHN) and Long Short Term Memory (LSTM). In general, these approaches are formalized as machine learning regression problems.
The input of software effort estimation for each software issue is the title and description of issues. The output is the number of story points that usually ranged from 1 to 100. According to [3], the value of story point should be in the modified version of Fibonacci numbers to ensure that the different story points are not too close to each other. Since then, the distribution of story points is range from 1 to 100 but not in continuous uniform distribution. The input of this problem can be considered as a type of software documentation that is used for requirement collection.
Since GNN had been applied in several research works in SE, in our knowledge it is a new area to apply GNN techniques on software requirement datasets. In this work, we want to study the potential and challenges when adapting a GNN model in text classification for story points categorization. We split the story points label in the dataset [1] into four levels. Next, we build the training model using a technique called Text Level GNN [4]. This training model is used for story points level prediction. We published our code and dataset at this site 1 .

II. RELATED WORK
In the problem that we considered as text classification in SE, there are several approaches to applying GNN for this problem. The first approach is proposed in [5]. In this work, there are two types of nodes constructed from the graph. They are nodes as documents and nodes as words inside documents. The edge representations are in one of these types: the relationship between two documents with the edge weight as the similarity between documents, the relationship between words represented by the co-occurrence of them in documents, the relationship between words and documents by the TF-IDF score. The graph is constructed from all the documents in the training dataset. The second approach is called text level graph in NLP [4], which worked by extracting the graph for each input text or document. Compared to [5], [4] is considered as an improvement approach since it can construct a graph for each document that consumes less memory and be feasible to interact with unseen test data. In our work, we inherit the model from [4] for our problem.

III. APPROACH
In this project, we try to solve the story point estimation problem by inheriting the Text Graph Neural Network proposed by [4]. We call our solution as TextLevelGNN-StoryPointEstimator.The overview architecture of our approach can be shown in Figure 1.
The target object given to each issue is the level of story points. According to the distribution of software effort value,  Table I.The process of issue extraction can be illustrated as follows. First, a text combination of title and description will be concatenated. Second, a graph will be constructed for each issue, which reflects the relationship between each word and its adjacent words. Finally, a GNN model is used to train a model which can infer the software effort levels. Next, we discuss important modules provided in our approach.

A. Graph Construction
Each issue contains two textual information: the issue's title and the issue's description. We provide a combination of title and description as the textual representation of a software issue. There are several types of graph construction from the text. For this problem, we inherit the graph construction from [4]. The idea of graph generation is to make an edge from a word to an offset of adjacent words in a sequence. There can be the case that there are edges that appeared multiple times between two specific words. Those cases will be reflected by the edge weight that highlights the popularity of co-occurrence between words. Edges that appeared rarely between words are considered as public edges. The offset of neighbor words between a specific word can be called the sliding window offset w. The sliding window offset is larger, the number of edges is increased, causing the increment of the complexity of the graph.

B. Graph Neural Network
One important element to construct a graph neural network is the process of passing information between nodes and edges inside the graph. This mechanism is called Message Passing Mechanism (MPM). In this problem, TextLevelGNN [4] learns the representation for each node by its neighbor and then combines the representations of all nodes/ words to predict the label. The initial vector for each node is retrieved by the vocabulary of Glove vector representation.
[6]. The combination mechanism to predict the label can be done by the softmax and Relu functions [4].

A. Dataset and Configurations
We use the dataset collected from [1]. There are 16 software projects developed by Agile development, which contains over 23000 software issues and actual story points. Similar to [1], we split the dataset as 80% for training and validation, 20% for testing. We use the same configuration with prior work [4], with batch size as 32, drop out probability as 0.5, sliding window offset as 20, and the default Glove representation pretrained model with 300-dimensional vectors for each word in English. The distribution of the level of story points and numbers of issues per each level are shown in Table I. We select the range of story point levels based on the median of story points for each project on dataset [1].
We compare the GNN approach with the traditional approach of story point level classification using TFIDF vectorization and Random Forest for classification. For TFIDF vectorization, we use the n-gram as the combination from 1gram to 4-grams and use the library TfIdfVectorizer in Python [7]. For RandomForest (RF) classification, we use its implementation from Scikit-learn library [8]. We call the traditional approach TFIDF-RF. We aim to answer the following research questions (RQs): 1) RQ 1. Accuracy of TextLevel GNN classification from raw text of issues' titles and description. 2) RQ 2. Analysis on ability of TextLevel GNN for regression for story points estimation. 3) RQ 3. Analysis on abilities of optimization in running time and graph complexity for TextLevel GNN.  The result of TextLevelGNN is shown in Table II. From the results on 16 projects, we achieve an accuracy as close as the traditional approach for classification. We achieve the highest accuracy on the bamboo project as 100% while the lowest accuracy is the aptanastudio project as over 41%. We achieve high accuracy as TFIDF and Random Forest in almost all projects. The project that TFIDF and RF outperform us most are mule as about 9% while we achieve significantly higher accuracy on the talendataquality by 3%. We analyze the data inside 16 projects. We see that the mule and talenddataquality provide challenges with low accuracy in story point regression in prior work [1], which our experiment has consistent results in our classification problem.

C. RQ 2. From Text Classification to Text Regression
The TextLevelGNN approach [4] is applicable in text or issue classification. Though the space of story point value is not in uniform distribution, the story point estimation should be considered as the text regression problem instead of classification. We study other works in Software Engineering that applied GNN. We see that most of them focus on the classification or code generation instead of regression such as [9]. There are a few problems in SE that can be formalized as regression problems such as project assignment scoring [10] and Github star project prediction [11], however, none of them applied GNN as solutions. We studied several works on graph regression in NLP and see that most of them focus on datasets in different research areas such as image processing [12].
Considering story points as labels for classification A simple solution to convert from classification to regression problem is that we consider the story points themselves as labels for classification. Next, we evaluate the correctness   Table II, reveals the fact that the TextLevelGNN approach achieves higher MAE than the BOW-RFR approach. It means the TextLevel-GNN didn't perform as well as the traditional approach with this simple solution for converting from classification to regression problem. In the upcoming work, we intend to change the mechanism in [4] to regression learning to support story point estimation.

D. RQ 3. Optimization in Running Time and Graph Complexity
We see that the running time in the training of your approach is longer than the traditional work. The BOW-RF approach requires 4 minutes to complete the training step, while the TextLevelGNN requires about 25 minutes for the same tasks of training on 16 projects of [1]. The testing step achieves almost the same running time for the TFIDF-RF approach and our approach. We study the result and see that one problem due to long training time is the number of edges in our graph is usually high, which can be more than 500000 edges in complicated projects such as memos. In future work, we can optimize the running time by simplifying the graph in which we highlight important information. First, along with making edges between adjacent words, we can filter and make edges between important words instead. Second, we can build the graph based on the information from the part of speech tagger to avoid non-necessary information.
1) Analysis on the scale of graph.: To study how the complexity can affect the performance of the training process, we analyze the nodes, the edges, the size (number of issues  for training) and compare with similar metrics on the popular natural text dataset R8 which was used in the work [4].
We have some observations. First, the size of 16 projects is smaller than the R8 dataset. The number of issues for each project ranges from over 200 to 3000. Second, the number of nodes in the R8 dataset is about 3000, which is lower than some projects like datamanagement, memos, or talendesb. It means that there are more new words inside each issue instead of a standard NLP text dataset. We study some issues and we see that the description of issues is usually longer than the description that the combination of title and descriptions can be more than 10 sentences. This fact of causing the risk of out-of-vocabulary words that didn't appear in the standard vector corpus Glove used in [4]. We can improve this challenge by building a new Glove representation trained from the context of issues' description. Third, the number of edges is much larger than the number of nodes. This is because the mechanism of making edges between a word and neighbor words in the offset w = 20 causes the exponential of the edges. Similarly, the number of edges provided in the R8 dataset is high which is about 1.3 million edges. In the issues dataset, the project that has the highest number of edges is memos project, due to the complexity of long issues' description. The project that has the least number of edges is the jirasof tware project, which can be due to the compact number of training records. Forth, for the running time, the running time for each project ranges from about 9 seconds to 370 seconds, which is comparable with the R8 dataset.
2) Simplifying the graph of issues by verb and noun.: We analyze the potential of simplifying the input text as an issue by filtering its words' roles in part of speech tagging by an experiment that its results are shown in Table VI and  Table V. In this study, we filter the words that are verbs and nouns and remove all words in other tags of the issue. We run the experiment on the datamanagement project, we called this configuration verbN ounF ilter. The results show that the accuracy is 56.36% on this project, compared to 61.72% with the normal text without the filter. Moreover, the training time and the graph complexity are alleviated. Though this is a very beginning and heuristic strategy to simplify the textual content, it shows the potential of simplifying the graph that focused on important parts of speech tags. The total average accuracy on 16 projects was raised to 79.66%.
3) Sensitivity on window size.: We study the impact of window size on the number of edges and the classification accuracy by the sensitivity experiment by changing window size to one of the following sizes: [2,5,10,20,50,100]. We evaluate projects jirasof tware and clover. The impact of window size on the number of edges is shown in Table III, which shows the high increase of edges when we increase the window size. The accuracy experiment is shown in Figure 2. We can see that too big a window size can cause a decrease in the accuracy, possibly due to many non-useful edges of the constructed graph.
V. CONCLUSION In this project, we propose a new mechanism of story point level estimation. Instead of converting text sequence to vector representation for features, we apply graph as the data structure for story points level classification. By the experiment, we see that the TextLevelGNN approach achieves comparable accuracy compared to the traditional approach using TFIDF for vectorization combined with Random Forest for machine learning classification. There are rooms for improvement in future works, which is related to adjusting the classification problem to the regression problem and optimizing the running time by filtering important information of the input text for graph construction.