{'fraction_non_alphanumeric': 0.034820236987907954, 'fraction_numerical': 0.011465199983823351, 'mean_word_length': 4.943396226415095, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Generating Natural Language Requirements based on Domain Ontology

Software requirements specification is undoubtedly critical for the whole software life-cycle. Nowadays, writing software requirements specifications primarily depends on human work. Although massive studies have been proposed to fasten the process via proposing advanced elicitation and analysis techniques, it is still a time-consuming and error-prone task that needs to take domain knowledge and business information into consideration. In this paper, we propose an approach, named ReqGen, which can provide recommendations by automatically generating natural language requirements specifications based on certain given keywords. Specifically, ReqGen consists of three critical steps. First, keywords-oriented knowledge is selected from domain ontology and is injected to the basic Unified pre-trained Language Model (UniLM) for domain fine-tuning. Second, a copy mechanism is integrated to ensure the occurrence of keywords in the generated statements. Finally, a requirement syntax constrained decoding is designed to close the semantic and syntax distance between the candidate and reference specifications. Experiments on two public datasets from different groups and domains show that ReqGen outperforms six popular natural language generation approaches with respect to the hard constraint of keywords(phrases) inclusion, BLEU, ROUGE and syntax compliance. We believe that ReqGen can promote the efficiency and intelligence of specifying software requirements.


I. INTRODUCTION
There is no doubt about the importance of software requirements to the whole software life cycle [1][2] [3]. As the vital product of requirement analysis, software requirement specifications act as the essential bridge between the requirement analysis stage and the following development and test. Nowadays, writing software requirements specifications primarily relies on human work, and this work is complex and time-consuming due to the following factors.
• Enough domain knowledge is required for stating the right content as well as selecting appropriate words and expressions. But domain analysis typically requires non-trivial human effort [4]. • Writing the specifications word-by-word is time-consuming, let alone that many expressions are repeated, especially in similar or related requirements. Writing or locating, copying, and then pasting the repeated content is a waste of human effort. • Generally accepted requirement syntax, such as EARS [5], is suggested for writing well-formed specifications. Learning and carefully applying non-business-related knowledge is also a burden on requirement analysts. Intuitively, it would be quite helpful to automatically recommend requirements specifications as long as the analysts can provide some simple related information almost effortlessly. To the best of our knowledge, most of the automated requirements specifications generation work focus on transforming software engineering models (e.g., i* framework [6] [7], KAOS [8] [9] [10], UML models [11] [12] [13]) or other semi-structured inputs (e.g., security goals in specific syntactic patterns [14]) into natural language requirement specifications based on pre-defined rules, which are usually brittle and restrict the usability of these approaches. What's more, constructing expressive and precise models is another complex work too.
In this work, we aim to automatically generate requirement statements once the analyst has an intuitive idea and can provide two or more keywords (phrases) of the desired requirements. Optionally, the analyst can also suggest the syntax roles of part or all keywords. We hope to recommend the requirement specifications although requirement analysts probably need to revise our generation for the final acceptance. To illustrate our target task, we contrived one example from a requirement instance of ISO/IEC/IEEE 29148:2018(E) [33] shown in Fig. 1.

Input: Signal [condition], system [subject]
Target: When signal x is received, the system shall set the signal x received bit within 2 seconds.
Note: the syntax roles in brackets are not mandatory. In this paper, we propose an approach, named ReqGen, to generate requirements specifications from the keywords (phrases) provided by requirement analysts. In particular, three critical designs are proposed upon the basic Unified pretrained Language Model (UniLM) [34], which is selected as the backbone of our framework because of its promising performance on natural language generation (NLG) tasks [35]. First, we inject domain knowledge in the format of pseudosentences created from domain ontologies into several selected layers of UniLM. Second, we use a copy mechanism in the training phase and set the given keywords as a hard constraint to ensure that they occur in the final statements. Last but not least, we design a requirement syntax constrained decoding approach to accommodate to the requirement syntax constraints of the generated statements, given that well-formed requirements should follow a certain syntax [36] [5][33].
To evaluate our approach, we conduct experiments on public datasets consisting of two domains, which were collected from previous work [37]. The results demonstrate the promising performance of ReqGen, especially on the dataset with a larger knowledge scale. The comparisons with six popular NLG approaches show that ReqGen obtains better Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and requirement syntax compliance. We also show the effectiveness of the three proposed components in ReqGen using an ablation experiment. The major contributions of this work are as follows.
• An approach ReqGen of automatically generating software requirement statements based on two or more keywords. • The evaluation of ReqGen on two public datasets from different domains with promising results, and suggestions on knowledge injection to pre-trained models. • The source code and experimental data publicly available on Github https://github.com/ZacharyZhao55/ReqGen. Significance. Requirements recommendation is a major concern in RE. Comparing with the traditional approaches, we take a step forward by recommending the final requirements specifications rather than only identifying the useful but indirect information.

II. BACKGROUND
This section describes three key techniques: UniLM, attention mechanisms and bidirectional long short-term memory (Bi-LSTM).
1) UniLM: We use UniLM [34] as the backbone of our approach because of its good NLG performance [35] [38]. UniLM is a multi-layer Transformer network with 786-hidden dimensions, and 16 attention heads. Its parameter size is 340M, and the activation function is Gaussian Error Linear Unit (GeLU), the same as those of Bidirectional Encoder Representation from Transformers(BERT) [39]. It is pretrained using two unidirectional language models (LM), one bidirectional LM, and one seq2seq LM. For our requirement generation task, we configured it using seq2seq LM (also known as the encoder-decoder model). UniLM uses the 12layer bidirectional Transformer encoder like BERT. Each encoder layer includes a multi-head attention and a feed-forward neural network. During decoding, UniLM uses beam search (beam size is 5 in our implementation) to select the candidate tokens of top-k scores in each step.
2) Attention mechanism: The model with conventional encoders cannot pay attention to information outside the input sequence. Our aim is to make the model can pay more attention to the injected knowledge; in other words, to let the model change the weights of the original hidden state so that higher weights can be assigned to the injected knowledge.
Many attention methods exist, including the dot product model, scaled dot product model, additive model, and bilinear model. Moreover, the steps of these methods are almost the same, including (1) calculation of the similarity between Query and Key, and obtaining the weights; (2) normalization of the weights; and (3) summation of the weights with Value. The scaled dot product is the most-common, fast, and space-efficient attention mechanism [40], and we use it. This model has an input consisting of a Query and Key with d k dimensions, and a Value with d v dimensions. We first calculate the dot product of the Query and Key (MatMul), then divide every Key by √ d k (Scale). Finally, we feed it to a softmax layer to obtain the weights corresponding to the Values.
3) Bi-LSTM: To better understand the injected knowledge, contextual information should be considered in the model. Before injection, we need to encode the knowledge. In the method proposed in this paper, we use Bi-LSTM [41] to encode the knowledge in order to capture its contextual information from the injected knowledge, where the knowledge is embedded by BERT. Bi-LSTM is a variant of Recurrent Neural Network, which combines two standard LSTM [42] layers in opposite directions to learn the two-way representation. An LSTM cell has three gates: the input, forget, and output gates. These three gates have different functions, acting as filters that decide which information to keep and which to forget. This increases the accessible information of the Bi-LSTM network and hence the model can better understand the context of the knowledge.
III. OUR APPROACH: REQGEN As shown in Fig. 2, we select UniLM as the backbone of ReqGen, because of its advantages in natural language understanding and NLG [34]. On the basis of UniLM, four components are designed and implemented to improve the compliance of the generated statements with domain knowledge and software requirements syntax (as indicated by the light blue rectangles in Fig. 2): • A knowledge preparation module that retrieves keywordrelated information from the domain ontology (in terms of triples consisting of entity pairs and their relationships), transforms all related information into pseudo-sentence, and selects the knowledge to be injected into the different layers of UniLM. • A knowledge-injection model to inject the knowledge produced by the knowledge preparation module into UniLM. The injected knowledge sequence is encoded by a Bi-LSTM structure using a BERT-based sentence embedding and is injected through the attention mechanism.
• A keywords copy mechanism is added in the original UniLM, which enables it to perform a copied-word classification task in the UniLM training. Moreover, a prediction method is added to the UniLM inference model to decide whether the next token is a copied word. • A requirement syntax constrained decoding module that includes a semantic-related metric used in the inference model with the original beam search to optimize the final statements towards a specific syntax.

A. Knowledge preparation module
The purpose of this module is to produce the knowledge to be injected in the backbone UniLM according to the input keywords from the domain ontology. It consists of three steps: multi-hop knowledge search, pseudo-sentence construction, and knowledge selection for the UniLM layers.
1) Multi-hop knowledge search: This step aims to obtain the keyword-related concepts and their relationships from a domain ontology. We would like to increase the probability of the co-occurrence of these concepts and the input keywords by managing the attention in the knowledge injection model.
We obtain knowledge from the domain ontology, which is a collection of all of the relevant concepts and their relationships in a single domain. It is represented as a graph structure composed of triples Entity 1 , Relation, Entity 2 , built based on OWL ontology language rules [43].
To obtain the context information of the keywords, we first perform a multi-hop graph search to acquire as much useful information as possible (five hops in this work), starting from the keywords, and collecting all types of entities in the retrieved paths.
Because all of the input keywords are regarded as start nodes, there must be repeated retrieved paths. We believe the concepts in the repeated paths are important; that is, the more often they are retrieved, the more important they are. Following this principle, we filter some concepts according to the retrieval times to avoid noise.
2) Pseudo-sentence construction: We convert the extracted multi-hops Entity 1 , Relation, Entity 2 into pseudo-sentences for the following injection task. In the OWL ontology language, entity types include Classes, Object Properties, and Named Individuals. And there are two types of relations between these entities: the subclass relation and constraint relation. We set pseudo-sentence generation rules for these two relation types as follows.
• subClassOf or hasSuperClasses: For the triple of a, subClassOf, b , we create a sentence of "a is subclass of b". And we convert the triple of a, hasSuperClasses, b into "a has super class b." • subPropertyOf : For the triple of a, subPropertyOf, b , we create a sentence like "a is subproperty of b.".
(2) The Constraint type includes has domain and has range relations.
• has domain: Specifies the domain of a property P, indicating that any resource with a given property is an instance of the domain class (e.g., teaching, hasdomain, teacher ). • has range: Specifies the range of a property P, indicating that the value of a property is an instance of the range class (e.g., teaching, hasrange, lesson ). Generally, the relationships of has domain and has range are paired. If the property has a has domain triple, there must be a corresponding has range triple simultaneously. Hence, we produce sentences for the relationship pairs. Taking the triples of teaching, hasdomain, teacher and teaching, hasrange, lesson as examples, we can create the pseudo-sentence "Teacher is teaching lesson." Note that the grammar of the pseudo-sentences generated automatically may be wrong. However, it can provide an important context for the concepts in sentences that is required by Bi-LSTM encoder.
3) Knowledge selection towards the UniLM layers: In the human learning process, when learning new stuff, we typically learn relatively broad knowledge first and then pay more attention to the essential parts, which is a repeated and gradual process. Inspired by this, we inject different keyword-related knowledge three times to the UniLM encoder and the injected knowledge is more refined every time, rather than the common one-time knowledge injection into the pre-trained model [38].
Following the traditional one-time knowledge injection in the first layer of the pre-trained BERT model [38], we observed that the injected knowledge cannot be reflected in the decoder of BERT. We believe the primary reason is the strong fitting ability of BERT on training data, i.e., too small weights of the external knowledge injected once than those of training data.
Therefore, we design a knowledge-injection mechanism by increasing the number of injections and selecting different knowledge for different layers of UniLM. In this work, we inject the knowledge into the 1 st , 2 nd , and 4 th layer of the UniLM encoder, as shown in Fig.2. Moreover, we inject more valuable information into the higher layers. In particular, we select all 5-hop keyword-related information for the first layer, 2-hop information for the second layer, and only 1-hop information for the fourth layer. This gradual refinement of the injected knowledge should enhance the ability of UniLM to absorb externally injected information. The evaluation and analysis of these three layers selection (i.e., 1,2,4) are shown in Section IV-B2.

B. Knowledge-injection model
As Fig. 2 shows, the knowledge-injection model is composed of three layers. The multi-hop information in the format of pseudo-sentences is first embedded using BERT and then encoded using Bi-LSTM. Finally, an attention layer is used to increase the weights of the injected knowledge in the hidden representation layer of the UniLM encoder.
1) Pseudo-sentence embedding and encoding: The basic unit of BERT embedding is a token. Thus, during the embedding phase, a pseudo-sentence set (knowledge) is first given to the BERT tokenizer. are added at the beginning and end of each sentence, respectively. Moreover, the tokenizer separates words into sub-words including the root of the words (e.g., "flying" is divided into ["fly","##ing"]).
There are three parts of the BERT embedding process: token embedding, position embedding, and segment embedding. Token embedding converts the tokens into vector representations in 768 dimensions ([1, n, 768], where n is the length of a sequence). Segment embedding is used to distinguish between the keywords and target requirements in each input pair ([1,n,768]). Position embedding converts the position information in the sequence into a vector representation of 768 dimensions ([1, n, 768]). Then we sum these three embeddings and use this representation as the output of the BERT embedding.
We send the embedding to Bi-LSTM encoder [41] to obtain the pseudo-sentence encodings. The shape of the encoded hidden representation is [b,n,768], where b is the batch size of UniLM, and n is the sequence length.
2) Attention mechanism: We use an attention mechanism to emphasize the importance of the injected knowledge. An attention mechanism is formally defined as follows: Here, Q is a Query, which represents the hidden representation of the UniLM encoder layer; K and V are the Key and Value, respectively representing the hidden representations of the pseudo-sentences(knowledge) encoding. To obtain the attention weights of Q and K, we use the scaled dot product method. M is the joint mask of the pseudo-sentence and the original input of UniLM (i.e., source keywords and target specifications) and d represents the length of the last dimension of K. According to experience, the use of √ d reduces the sensitivity of the method to the length of K, and improves the stability of network training. Then, we calculate the dot product between A and V to update V and obtain the context representation of the attention. Finally, we sum the updated values H ctxt and H U niLM , and feed them into LayerNorm (i.e., normalization layer) to obtain the hidden representation of the knowledge.

C. UniLM module
We modify UniLM in three ways, by adding multi-layer knowledge injection, a copy mechanism in the training model, and a requirement syntax constrained decoding in the inference model. The knowledge injection was discussed in Sections III-A and III-B. Here, we focus on the other two modifications.
1) Copy mechanism: Copy mechanism is very popular in NLG tasks such as translation and conversation generation [44]. Its aim is to resolve the "hard-constraint" problem, which requires the tokens or fragments of the source sequence to occur in the target sequence. However, most copy mechanisms can only guarantee the copy of a single word, not continuous fragments [45]. But in our scenario, the input keywords (phrases), even the single-term words, can be multi-term words after tokenization, making it a challenge to ensure the integrity of words in the generated statements.
We adopt a simple but novel copy mechanism [45]. In our implementation, this mechanism first marks the copied fragments in the target requirements according to the source keywords. Then, we add a new copied label prediction task in the training phase of our seq2seq model to predict whether a token has been copied from the source keywords or not. In the decoder prediction stage, the original next-token prediction probability is changed to a mixed probability model of the next token prediction probability and copied token prediction probability.
2) Requirement syntax-constrained decoding: Well-formed requirement specifications should follow a certain syntax, and there are several related guidelines or models, such as EARS [5], IEEE 29148:2018 [33] and M-FRDL [36]. They all define the fine-grained elements of single requirement specifications and each indicates one semantic role.
The requirements generated by ReqGen should attempt to follow at least one existing syntax so that they can accurately capture stakeholder needs [33]. Hence, if requirement analysts follow specific requirement syntax when writing requirement specifications and they could give the semantic roles that the keywords belong to (the content in brackets of Fig.  1), we would like to use this information to improve the generated sentence further. We designed an indicator called RS4RE to evaluate the overlap of the semantic constitution of the generated requirements with those set by engineers. The RS4RE value of each generated candidate statement is used to help select the final statement by adding it to the original probability score during beam search.
RS4RE is defined formally as follows. We measure the closeness of agreement of each semantic element in the generated statement and the input of an analyst.
Let N be the number of semantic elements in the syntax requirement analysts have selected and E i be the i th element.
Here, E i ref is used to indicate the word set of the element E i given by the requirements analysts, and E i (R aut ) is to indicate the set of words of the element E i in the automatically generated sentence. The agreement on element i is calculated as the ratio of the scale of overlapping words to the size of the manually given set. Moreover, α i is a hyper-parameter that indicates the weight of element i and the sum of all α i is 1.
In this work, we use M-FRDL [36] as the syntax because they shared the source code with us. This enables us to automatically identify the fine-grained elements in the natural language requirements, which is required by the automated calculation of RS4RE.  [37].

IV. EXPERIMENTAL
The UAV requirements are from the University of Notre Dame, 1 including 99 requirements [46]. And the public ontology for the UAV domain 2 includes 400 entities. The BAS requirements are from the Standard BAS Specification (2015) [47] and consist of 456 requirements, involving functional, performance and security requirements. And the open-domain model of BAS 3 includes 484 entities.
The requirements are represented using natural language sentences, and we required the related keywords before the model training and testing. Thus, we performed a reverse extraction process. To be specific, we automatically extracted the noun words or phrases based on the parts-of-speech tagged by Stanford CoreNLP [48]. We extracted the noun and noun phrases with a series of linguistic filters for the nested noun selection, such as N oun + N oun. For each requirement, we randomly selected n noun phrases (n ∈ [2, N ] where N is the total number of noun and noun phrases in one requirement) as the keywords.
To evaluate the effectiveness of the requirement syntax constrained decoding described in Section III-C2, we implemented the M-FRDL constraint [36] and manually assigned the semantic roles to the randomly selected keywords according to their context in requirement statements. To ensure correctness, we invited one author of the work [36] to check and revise them.
Another problem with the data is that, the domain ontology may be unable to cover the all content of the requirements because these dataset were generated by different groups who have different concerns. Only if the keywords are contained in the domain ontology, will the ontology be helpful for the keyword-driven requirements generation. Thus, we performed automated domain ontology completion using an approach of [49]. This approach first aligns the requirements and domain ontology using TransE [50]. Then it selects the requirement concepts, that are related to the entities in the domain ontology, and adds them as well as their corresponding relationships to the ontology.
Given the limited data, we performed 10-fold and 5-fold cross-validation on the UAV and BAS requirements respectively. In addition, we performed a batch knowledge injection and injected the knowledge for all of the random keywords of the testing requirements, considering the potential association of the requirements. In particular, 17,681 pseudo-sentences were created for the 10 test requirements of UAV, and 139,494 pseudo-sentences for the 93 test requirements of BAS.
2) Baselines: We selected six popular constrained text generation approaches as the baseline methods, including four pre-trained models and two other NLG models (i.e.,POINTER [51] and CGMH [52]). Similarly, we trained these models using the UAV and BAS dataset separately (10-fold and 5fold cross-validation, respectively).
• BERT [39] jointly conditions on contextual information, which enables pre-training and deep bidirectional representation from unlabeled text. • Generative Pre-trained Transformer 3(GPT3) [53] uses the one-way language model training of GPT2. The model size was increased to 175 billion and 45 terabytes of data were used for training. GPT3 can perform downstream tasks without fine-tuning in a zero-shot setting. We did not perform fine-tuning neither. • Bidirectional and Auto-Regressive Transformers (BART) [54] is a denoising autoencoder built with a sequence-tosequence model suitable for various tasks. It uses a standard Transformer-based neural machine translation architecture. It is trained using text corrupted with an arbitrary noising function and by learning to reconstruct the original text. • UniLM [34] uses three types of language modeling tasks for pre-training, which is achieved by employing a shared Transformer network and using specific self-attention masks to control the context of the prediction conditions. • Zhang et al. [51] proposed POINTER, which is based on the inserted non-autoregressive pre-training method. A beam search method was proposed to achieve log-level nonautoregressive generation. • The Constrained Generation by Metropolis-Hastings sampling (CGMH) method [52] can cope with both hard and soft constraints during sentence generation. Different from the traditional latent space usage, it directly samples from the sentence space using the Metropolis-Hastings sampling.
3) Metrics: We selected BLEU [55] as the first metric, which is commonly used in machine translation, NLG and source code generation. It measures the degree of overlap between the generated and reference sentences using n-grams. The higher the degree of overlap, the higher the quality of the generated text. Here, BLEU1 measures word-level accuracy and BLEU2 measures sentence fluency to a specific degree. In addition, we employed the ROUGE metric [56]. ROUGE-N calculates the total sum of the number of n-grams occurring in both the generated and target sentences, and ROUGE-L calculates the longest common subsequence. We calculated recall, precision and F-measure for each kind of ROUGE rather than the simple n-gram recall [56].
Besides, we used the RS4RE metric to evaluate the agreement of syntax compliance between the generated and target sentences, as described in Section III-C2. Here, E i ref refers to the set of words in the i th semantic element of the target requirement. We also recorded the time used by each model (except GPT-3) for training and testing, considering its practical value.

B. Experimental results and analysis
1) RQ1 effectiveness of our approach: We illustrate the experimental results for UAV and BAS cases in Table I, and   Table II presents an example of the requirement generated by each baseline and ReqGen, from which we can draw three conclusions: (1) Our method achieves the best or second-best results for all metrics. In the UAV case, our method yields the best results for five metrics and the second-best results for the remaining eight metrics. BART achieves good performance with six best and five second-best results, which demonstrates the benefits of its hard constraint design. However, it can only embed a single input word and loses the complete semantic meaning of phrases (i.e., it obtains better BLEU1 and ROUGE-1 results, but weaker BLEU2 and ROUGE-2 results), which can also be seen in the example in Table II. Moreover, ReqGen obtains a better RS4RE result than BART, indicating that it has a better semantic-oriented text generation ability.
On the BAS domain, our ReqGen achieves seven best and four second-best results, outperforming all baselines. The basic UniLM performs moderately well because it considers both the input and its context during the next token prediction. This is the reason that we select it as the backbone of ReqGen.
(2) The four pre-trained models perform better than the other two baselines. Among the four pre-trained models in the UAV domain, BART obtains the best ROUGE-1, ROUGE-2, and ROUGE-L results. It even outperforms ReqGen in ROUGE-1, which is possible because BART forces all keywords to be included in the output during decoding. However, because it only sees the input, it performs worse in ROUGE-2. UniLM obtains good BLEU1 and BLEU2 results and the best ROUGE-2 results in the BAS case, indicating its better ability to generate relatively fluent statements.
CGMH performs worst according to the two BLEU metrics and the F-measure of the three ROUGE metrics, as shown in Tables I and II. POINTER also has a weak performance, and a possible reason for this performance could be that it places the entire generation burden on the decoder, which means that the sentences it generates are very long, but the correlation with the target is weak (see the example in Table II).
(3) Our method is slightly slower than UniLM and BART. As shown in the last column of Table I, we recorded the average time consumed by each method (in hours) to perform the 10-fold cross-validation on the UAV and 5-fold cross-validation on the BAS data. We did not record the time used by GPT3 because it does not need extra fine-tuning [53]. We observe that ReqGen needs slightly more time than UniLM and BART, but less time than the other approaches, even although it uses extra knowledge injection, a copy mechanism, and requirement syntax constrained decoding.
2) RQ2 effectiveness of multi-layer knowledge injection: We also evaluated the effectiveness of the multi-layer knowledge injection in the two domains. To implement and evaluate ReqGen, given that we collected 5-hop keyword-related information and there are 12 layers in UniLM, we had to determine 1) which levels should have knowledge injected, and 2) what kind of knowledge should be injected into the different layers. Because of the uncertainty in the injected layers and the injected knowledge for one specific layer (i.e., which hop  When a UAV is loaded the internal simulator shall display the location of the UAV. BART landing internal flight simulator shall compute the ground position of a UAV. CGMH simulator owners for takeoff at a corresponding latitude. POINTER in which a single point landing system, when a single point that is assigned a separate gps system or for an active navigation system has to orient at its current position and to display the current ground coordinates.

ReqGen
When a landing UAV is assigned to a UAV, the internal simulator shall compute the ground longitude latitude and latitude of the UAV.
in the five hops), there are many combinations that could have been used in our experiments. Thus, we pruned the candidate combinations for the experiments.
As for the injected layers, Li et al. [38] injected knowledge into the 1 st layer. Jawahar et al. [57] experimented with different layers of BERT on 10 sentence-level detection tasks. They observed that the 1 st and 2 nd layers learn surface information such as word detection in sentences. Layers 4 to 7 learn syntactic information such as word order sensitivity. Layers 8 to 12 learn semantic-level information such as subject-verb agreement. Inspired by these two studies, we focused on three combinations of shallow layers, shallow + syntactic layers, shallow + syntactic + semantic layers, with layer combinations of (1,2), (1,2,4) and (1,2,4,8), respectively. Moreover, considering the five hops in our collected information, we made an extra evaluation on the combination (1,2,4,8,11) and each layer was assigned 1-hop information.
For the injected knowledge, we followed the cognitive process of human learning, in which human usually start from general and overview knowledge and then pay close attention to critical parts. Similarly, we assigned more keywordknowledge in the lower layers. For example, for the layer combination (1,2), we assigned all 5-hop knowledge to the first layer and the most related 1-hop knowledge to the second layer. The injected knowledge details of different layer combinations can be found in the layer(hop) of Table III, which presents the results of this experiment.
For the UAV case, we observe that the best BLEU1 and BLEU2 scores are achieved by the (1,2,4) combination. For ROUGE-1, the best recall is achieved by injecting knowledge into layers 1, 2, 4 and 8, the best precision is achieved by the (1,2) combination, and the best of F-measure is achieved by the (1,2,4) combination. Similar phenomenon are observed for the ROUGE-2 and ROUGE-L metrics. In the BAS case, the best BLEU1 and BLEU2 results are also achieved by the (1,2,4) combination. For ROUGE-1, the best F-value is achieved by the (1,2) combination; however, for ROUGE-2 and ROUGE-L, the best performance is achieved by the (1,2,4) combination.
In summary, the (1,2,4) combination with the 5-, 2-, and 1-hop knowledge injection achieves seven best results in the UAV case, and nine best results in the BAS case, out of all 11 metrics. Hence, we suggest three knowledge injections by injecting all 5-hop knowledge into the first layer, 2-hop knowledge into the second layer, and the most critical 1-hop knowledge into the fourth layer.
3) RQ3 effectiveness of frequency filtering in the knowledge search: We further evaluated the impact of different frequency sets in the knowledge search on the final requirement generation (as described in Section III-A1). We regarded more frequent knowledge as more important, and our aim was to inject only important information into ReqGen to reduce noise pollution in the data.
To evaluate the impact of a frequency filter (a single dependent variable), we set the injected layer as a constant. We experimented with frequency thresholds of 0, 10, and 50 on all 5-hop knowledge injected into the 1 st layer. In other words, we injected all traversed entities, entities occurring more than 10 times, or those occurring more than 50 times in the ontology graph traversal. The results are shown in Table IV and the best result of each metric is highlighted with bold font.
We observe different results for the two cases. For UAV, no frequency filtering is best, followed by 10-and 50-frequency filters. Whereas, for BAS, the best performance is achieved with the 10-frequency filter, and 50-frequency filter is the second best. This result leads us to the following two observations. (1) Intuitively, the frequency set is strongly correlated with the scale of injected knowledge. In our case, the 5hop knowledge of the UAV domain includes 17,681 pseudosentences and BAS includes 139,494 sentences (approximately 7.9 times). (2) When there is a massive amount of information, less and more refined knowledge is more valuable. For example, 50-frequency filter is better than no filter in the BAS domain. However, when the information is small, all related knowledge is valuable (e.g., in the UAV case, no filter is better than the 10-frequency filter, which is better than the 50-frequency filter). However, because of the limited available cases, we cannot give a criterion for frequency filter selection temporarily in this initial study. 4) RQ4 ablation experiment: The ablation experiment was designed to verify the effectiveness of each critical proposed component in our ReqGen: multi-layer injection (including frequency filtering), copy mechanism and syntax-constrained decoding. Table V shows the results of ReqGen and its four variants. For the convenience of comparison, we indicate the values, that are better than those in the previous row with an up-arrow.
In the UAV case, compared with the one layer knowledge injection, the three injections into layers of 1, 2 and 4 yield an approximately 1% increase in BLEU1 and BLEU2. For the ROUGE results, the improvement is primarily in the three precision scores, which are increased by 0.5%. However, the recall in ROUGE was slightly decreased, possibly due to some related knowledge loss in layers 2 and 4. However, the F-measure values remained unchanged or turn better with respect to the values obtained for the single-layer injection. In the BAS case, all metrics, including the BLEU and three ROUGE metrics, were improved by three-layer knowledge injection. This observation shows that the multi-layer injection design mainly improves the BLEU metric and the precision of ROUGE, indicating that this mechanism enhances the ratio of valid words in the final statements. In other words, although the single-layer injection can help the model to obtain the knowledge, the multi-layer injection is further helpful for knowledge absorption.
The comparisons between the second and third rows in both the UAV and BAS domains show the impact of 10-frequency filtering on layers of 1, 2, and 4. We can see that the two BLEU values in both cases are better with this frequency setting.
For UAV, the precision and F-measure of the three ROUGE metrics have been increased further. However, the recall values are weaker. This is expected because the filtering operation on a relatively small set of knowledge would probably lessen the number of useful clues, whereas, for BAS, due to its larger scale, all metrics have been improved even after filtering. We note that the 10-frequency filtering indeed effectively reduces the noise.  In this example, when no-frequency filtering is used, the terms "loses" and "routing" in the domain ontology are injected into ReqGen and reflected in the generated requirement, showing that noise words are reflected in the lexical constitutions of the generated requirements, which then determine the semantic meaning. By contrast, the example with 10frequency filtering shows that our frequency filtering method helps reducing the noise in the injected knowledge.
The comparisons between the fourth and fifth rows of Table V for both domains show the influence of the copy mechanism. From the results for the UAV domain, we observe that six results, mainly ROUGE scores, have been improved and achieve the best values with this design, whereas the two BLEU scores have fallen approximately 2%. Similarly, the results for five metrics have been increased and the remaining six are decreased for the BAS domain. This shows that the copy mechanism is helpful for the recall of the overlapping n-grams in the generated sentences. However, because it is designed to guarantee that the given keywords must appear in the generated sentence, this actually increases the length of the generated sentences. In other words, it is highly likely to dilute the ratio of the overlapping n-grams in the whole statements meanwhile, which decreases the BLEU scores.
We also show the influence of the copy mechanism with an example in Fig. 4, and we can see that all keywords, including the multi-term phrases, can occur in the final sentence. Without this mechanism, some keywords are abandoned, in favor of the more important injected knowledge and the syntax learned  during the pre-training process of UniLM.
Source: operator, trend log, setup information Target: Operator shall be able to change trend log setup information.
No copy mechanism: Operator shall be able to use a trend log to share trend logs with other operators.
Copy mechanism: Operator shall be able to use a trend log for quick trend log setup information.

Fig. 4: An example with or without copy mechanism
Finally, we added syntax-constrained decoding into the settings, and the results are listed in the fifth row of each case in Table V. We can see consistent enhancements of the ROUGE-2 results for both cases and a positive effect on the BLEU2 of the BAS domain, and no change to the BLEU2 result of the UAV domain. This indicates that our syntax decoding is helpful for valid 2-gram generation, the semantic units with a larger granularity than single-term words.
V. DISCUSSION 1) Threats to validity: Conclusion validity mainly concerns the evaluation metrics. We used BLEU1, BLEU2, and the precision, recall, and F-measure of ROUGE-1, ROUGE-2, and ROUGE-L, which are the most common metrics of NLG [35][38] [58]. Moreover, to consider the special syntax constraints of well-formed software requirements [5][33] [36], we designed the RS4RE metric, which measures the syntax distance between the automated generated and target requirements. We also recorded the training and testing time of the automated approaches for practical consideration.
Internal validity concerns the validity of the causal relations between the results and our approaches. We reused the open source code of UniLM, the backbone of our ReqGen and the six baselines to ensure the correctness of the implementation. Another threat comes from the measure of RS4RE, which involves the manual annotation of the semantic elements of M-FRDL [36] for the case requirements. In the two sets of requirements, the UAV requirements were annotated by the work of [36], and we used their original annotations directly. For the BAS domain, our authors performed pair annotations which were then carefully checked individually by the first author of [36]. We believe the annotations are trustworthy.
External validity concerns the generalization of the experimental results to other cases. In this initial study, we experimented with two public datasets from different domains with different scales. All of the requirements and ontologies are from different groups, reflecting different writing styles. However, because of the data limitations, we only performed 10-and 5-fold cross-validation on the UAV and BAS cases. Although the results are promising, more diverse data for further experiments will be needed in future.
2) Implications: Based on our results, the primary implication is that, like traditional large-scale text-based finetuning, concept-net knowledge injection is beneficial for pretrained models performing downstream tasks. This means that our approach is potentially helpful for other document generation, such as the description of software design. In addition, multiple knowledge injections into the pre-trained model are better than once, and the knowledge injected into higher layers should be more refined than that injected into the lower layers, just as in the human learning process. Our experiments show that injection of all 5-hop knowledge into the first layer, 2-hop knowledge into the second layer, and only the most relevant 1-hop knowledge into the fourth layer achieves best results on both domains in our task (as shown in Table III).
The second implication suggested by our study is that automated requirement generation requires related domain knowledge but "more" does not always mean "better". In the BAS case, 10-frequency filtering is better than no filtering. However, the injected knowledge is also critical because the general UniLM has to learn the domain knowledge for the valid requirement generation. Thus, for the small UAV case, no filtering is the best (as shown in Table IV).
3) Limitations: Temporarily, we only randomly selected a few keywords as the seeds of ReqGen for the requirement statement generation and did not restrict their roles (e.g., specifying that the subject is mandatory). We plan to explore the impact of syntax role configurations, that the given keywords belong to, to the specification generation.
Our method cannot replace the elicitation and analysis of users' requirements. In other words, our aim is only to assist engineers to quickly produce the requirements that they already roughly know (i.e., the keywords). Meanwhile, we cannot guarantee that all the specifications are generated with professional domain words and clear formal expressions.
We did not perform an empirical study of the practical usefulness of ReqGen in real-world practice. In this initial study, we only evaluated its effectiveness by comparing it with six popular NLG approaches on the common indicators of BLEU, ROUGE and our proposed RS4RE.

VI. RELATED WORK
We first review requirements generation research. Then we describe the related work about common NLG.
1) Automated requirements generation: The existing studies on automated requirements generation mainly focus on transforming the (semi-)structured models (e.g., business process model in [59][60], i* framework in [6] [7], KAOS and Ojectiver in [8][9] [10], UML models in [11][12] [13]) or other representation (e.g., security goals in [14]) into specific syntactic pattern-oriented natural language requirements specifications, based on a set of pre-defined rules. They usually require precise representation of the critical elements, such as the roles, inputs, outputs and their relationships in the business process model [59], or the security goals expressed as clauses with a main verb + several security criteria + several target assets [14]. Both the (semi-) structured inputs and the predefined rules restrict the application scope of these approaches.
Mohamed et al. [61] proposed to generate non-quality requirements based on grammatical rules and a supplied dictionary, to resolve small-scale requirements sets for current requirements quality checking work.
There are very few studies on requirements specification generation from a few simple keywords. Most available research on software requirements capturing and generation concerns automatically collecting and/or analyzing requirementsrelated information with the purpose of assisting requirements analysts to manually acquire, interpret, and specify the final requirement statements. This information can be requirement-relevant sentences obtained from domain documents [4][15] [16] [17][26] [31], features and their relationships mined from developer online chats [18], online reviews [23] [24] and product descriptions [19] [20], or the classifications on types of user statements [22][32] and on obligation or non-obligation statements in the contracts [31]. Besides, better product descriptions can be automatically generated with the techniques of summarization from user reviews [62][63] and website information [64]. Although these sources of information are definitely essential for requirements specification, analysts still need to invest significant time and effort interpreting this information and specifying the requirements clearly under the premise of knowing the basic requirements syntax and the domain phrases involved in the requirements.
2) Automated NLG with lexical constraints: In an early work on lexical constraints generation, Mou et al. [65] proposed the backward and forward language model(B/F-LM), and used recurrent neural network to generate previous and subsequent words conditioned on the given word. Liu et al. [66] extended the B/F-LM by introducing a discriminator, but these two methods can generate sentences with only one lexical constraint. To exceed this limitation, Hokamp et al. [67] incorporated constraints by performing a grid beam search in the sentence space. The CGMH framework [68] models local transitions (e.g., deletion and insertion) to achieve better fluency, but it is slow to converge. Sha et al. [58] proposed an unsupervised lexical constraint generation method, in which a series of differentiable loss functions are used to calculate the fluency of the generated sentences and to determine whether they satisfy the constraints. Ding et al. [69] proposed a framework to customize the message content for appealing different individuals.
Recently, the fine-tuning of pre-trained language models has provided more research opportunities for small datasets in many domains. BERT [39] and RoBERTa [70] use masked language modeling pre-training objectives for deep bidirectional representations by jointly conditioning on both the left and right contexts in all layers. GPT-2/3 [71] [72] and CTRL [73] are causal language models, which use auto-regressive language to model the target. UniLM [34] and GLM [74] combine the advantages of the first two models and use a special mask mechanism so that the tokens in the input sequence can focus on each other, whereas the tokens in the output sequence can focus only on the tokens to the left. MASS [75], T5 [76], and BART [77] are encoder-decoder models and adopt the standard Transformer structure [40].

VII. CONCLUSION
This study aimed to automatically generate requirement statements based on pre-defined keywords. We proposed an approach called ReqGen that fine-tunes UniLM by injecting keyword-related knowledge with a repeated emphasis on the most relevant ones, integrates a copy mechanism to ensure the hard constraint of keyword inclusion, and uses syntaxconstrained decoding to cater to syntax requirements. Com-pared with six popular baselines, we showed that ReqGen obtains superior performance on the requirements specification generation task.