{'fraction_non_alphanumeric': 0.036219204927211646, 'fraction_numerical': 0.009798432250839866, 'mean_word_length': 4.846358428805237, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Knowledge-Aware Procedural Text Understanding with Multi-Stage Training

We focus on the task of procedural text understanding, which aims to track entities' states and locations during a natural process. Although recent approaches have achieved substantial progress, they are far behind human performance. Two challenges, difficulty of commonsense reasoning and data insufficiency, still remain unsolved. In this paper, we propose a novel KnOwledge-Aware proceduraL text understAnding (KOALA) model, which leverages external knowledge sources to solve these issues. Specifically, we retrieve informative knowledge triples from ConceptNet and perform knowledge-aware reasoning while tracking the entities. Besides, we employ a multi-stage training schema which fine-tunes the BERT model over unlabeled data collected from Wikipedia before further fine-tuning it on the final model. Experimental results on two procedural text datasets, ProPara and Recipes, verify the effectiveness of the proposed methods, in which our model achieves state-of-the-art performance in comparison to various baselines.


Introduction
In this work, we focus on a challenging branch of machine reading comprehension (MRC), namely procedural text understanding. Different from traditional MRC tasks which ask questions about given documents (Rajpurkar et al., 2016;Lai et al., 2017), understanding procedural text requires AI models to track the participating entities throughout a natural process . Taking Figure 1 for example, given a paragraph describing the process of fossilization and an entity "bones", the target is to predict the state (not exist, exist, move, create or destroy) and location (a textspan from the paragraph) of the entity at each timestep. Such procedural texts usually include the comprehension of underlying dynamics of the Step 0 is used to identify entities' initial locations before the process.
process, thus impose higher requirements on the reasoning ability of MRC systems.
Recently, some approaches in procedural text understanding develop structured neural networks with global constraints and achieve competitive results (Das et al., 2019;Gupta and Durrett, 2019b;Amini et al., 2020). However, their results (~65 F1) are still far behind human performance (83.9 F1). Particularly, there are two major problems that still remain unsolved in this field.
First, as previous models mainly focus on improving global consistency and capturing process dynamics, they assume that the clues for making predictions have already existed in plain text, which does not hold sometimes. Not only do entities usually undergo implicit state changes, but their locations are also omitted in many cases, especially when humans can easily infer the location through commonsense reasoning. For instance, in Figure  1, the initial location of "bones" is hard to be directly inferred from plain text, unless the model is aware of extra commonsense knowledge "bones are parts of an animal". For statistical evidence, we find that an entity is not explicitly connected to its locations in 32% of the cases, and state changes (create/move/destroy) are not explicitly stated in 26% of the cases, by manually checking 50 instances from the popular ProPara dataset .
Second, data insufficiency hinders AI models from reaching their best performances. As fully annotated data are costly to collect, existing datasets are limited in size. The benchmark ProPara dataset only contains 488 paragraphs including 1.9k entities. Another recent dataset, Recipes , contains only 866 human-labeled paragraphs. Moreover, such paragraphs usually fail to provide sufficient information considering the complexity of scientific processes. For example, each paragraph in ProPara only contains~60 words on average (see Table 1 for more stats), which restricts it from describing a complex process in detail. Thus, data enrichment is in serious need on this task.
In this paper, we aim to address these two issues using external knowledge sources, namely Con-ceptNet and Wikipedia. To solve the challenge of commonsense reasoning, we perform knowledge infusion using ConceptNet (Speer et al., 2017). As a relational knowledge base composed of concepts and inter-concept relations, ConceptNet is naturally suitable for entity-centric tasks like procedural text understanding. An entity in our task can be matched to a concept-centric subgraph in ConceptNet, including its relations with neighboring concepts. Such information can be used as extra commonsense knowledge to help models understand the attributes and properties of an entity, which further provides clues for making predictions even if the answers are not directly mentioned in plain text. As shown in Figure 1, although it is hard to directly infer the initial location of "bones", we can find evidence (animal, HasA, bone) and (bone, IsA, part_of_animal) from the Con-ceptNet knowledge graph. Therefore, we propose to retrieve relevant knowledge triples from Con-ceptNet, and apply attentive knowledge infusion to the reasoning model.
As for the challenge of data insufficiency, we propose to enrich the training procedure using Wikipedia paragraphs based on text retrieval. Inspired by the great success of pre-training models (Devlin et al., 2019), we propose a multi-stage training schema for BERT encoders. Specifically, we simulate the writing style of procedural text to retrieve similar paragraphs from Wikipedia. Compared to existing datasets, such Wiki paragraphs are usually longer, more scientific procedural texts and contain more details about similar topics. We expect the BERT model learn to better encode procedural text through fine-tuning on this expanded procedural text corpus. Thus, we train the BERT encoder for an additional language modeling fine-tuning phase with modified masked language model (MLM) objective, before further fine-tuning the whole model on the target dataset. We also conduct a similar multi-stage training schema on ConceptNet knowledge modeling where we adopt another BERT encoder.
To sum up, we introduce our KnOwledge-Aware proceduraL text understAnding (KOALA) model, which incorporates commonsense knowledge from ConceptNet and is trained with a multi-stage schema. For evaluation, our main experiments on ProPara dataset show that KOALA reaches state-ofthe-art results. Besides, auxiliary experiments on Recipes dataset also demonstrate the advantage of our model over strong baselines. The ablation test and case study further show the effectiveness of the proposed methods, which makes KOALA a more knowledgeable procedural text "reader".

Related Work
Procedural Text Datasets Efforts have been made towards researches in procedural text understanding since the era of deep learning. Some earlier datasets include bAbI (Weston et al., 2016), SCoNE (Long et al., 2016) and ProcessBank (Berant et al., 2014). bAbI is a relatively simple dataset which simulates actors manipulating objects and interacting with each other, using machine-generated text. SCoNE aims to handle ellipsis and coreference within sequential actions over simulated environments. ProcessBank consists of text describing biological processes and asks questions about event ordering or argument dependencies.
In this paper, we mainly focus on ProPara , a more recent dataset containing paragraphs on a variety of natural processes. The goal is to track states and locations of the given entities at each timestep. Additionally, we also conduct experiments on Recipes dataset , which includes entity tracking in the cooking domain. These datasets are more challenging since AI models need to track the dynamic transitions of multiple entities throughout the process, instead of predicting the final state (SCoNE) or answer a single question (bAbI, ProcessBank). Besides, entities usually undergo implicit state changes and commonsense knowledge is often required in reasoning.
Procedural Text Understanding Models Our paper is mainly related to the lines of work on ProPara . ProStruct  applies VerbNet rulebase and Web search co-appearance to refine the probability space of entity state prediction. LACE (Du et al., 2019) introduces a consistency-biased training objective to improve label consistency among different paragraphs with the same topic. KG-MRC (Das et al., 2019) constructs knowledge graphs to dynamically store each entity's location and to assist location span prediction. NCET (Gupta and Durrett, 2019b) extracts candidate locations using part-of-speech rules from text paragraphs, and considers location prediction as a classification task over the candidate set. ET (Gupta and Durrett, 2019a) conducts analyses on the application of pre-trained BERT and GPT models on the sub-task of state tracking. XPAD  builds dependency graphs on ProPara dataset, which tries to explain the action dependencies within the events happened in a process. DYNAPRO (Amini et al., 2020) dynamically encodes procedural text through a BERTbased model to jointly identify entity attributes and transitions. In this paper, we aim at two main problems that have not been effectively solved by the above works: commonsense reasoning and data insufficiency. Benefiting from the commonsense knowledge in ConceptNet and the proposed multistage training schema, our model outperforms the aformentioned models on the ProPara dataset.

Commonsense in Language Understanding
Incorporating commonsense knowledge to facilitate language understanding is another related line of work. (Yang and Mitchell, 2017) infuses concepts from WordNet knowledge base with LSTM hidden states to assist information extraction. (Chen et al., 2018) proposes a knowledge-enriched co-attention model for natural language inference. (Lin et al., 2019) employs graph convolutional networks and path-based attention mechanism on knowledge graphs to answer commonsense-related questions. (Guan et al., 2019) applies multisource attention to connect hierarchical LSTMs with knowledge graphs for story ending generation. (Min et al., 2019) constructs relational graph using Wikipedia paragraphs to retrieve knowledge for open-domain QA. (Wang et al., 2020) injects factual and linguistic knowledge into language models by training multiple adapters independently. Inspired by previous works, we introduce commonsense knowledge from ConceptNet (Speer et al., 2017) into the procedural text understanding task, and prove that the retrieved knowledge contributes to the strong performance of our model.

Problem Definition
Here we define the task of Procedural Text Understanding. Given: , describing a process of T timesteps, e.g., photosynthesis or a cooking recipe.
• A set of N pre-given entities {e 1 , · · · , e N }, which are participants of the process.
• The entity's location at each timestep y l t (0 ≤ t ≤ T ), which should be a text span in the paragraph. A special '?' token indicates the entity's location is unknown. y l 0 denotes the initial location before the process begins.
Besides, the ground-truth location and state at timestep t are denoted as y l t and y s t , respectively.

Framework
The base framework of KOALA is built upon the previous state-of-the-art model NCET (Gupta and Durrett, 2019b), shown in Figure 2. Its major differences to NCET are the use of powerful BERT encoders, the knowledge-aware reasoning modules ( §4.2) and the multi-stage training procedure ( §5). Based on an encoder-decoder architecture, the model performs two sub-tasks in parallel: state tracking and location prediction.
roots absorb water from soil . water flows to the leaf . CO2 enters the leaf .  Figure 2: An overview of the KOALA model (left) & a detailed illustration of knowledge-aware reasoning modules (right), focusing on entity "water". Note that the location prediction modules are applied to each location candidate (root, soil, leaf, etc) in parallel, and perform classification among candidates at each timestep. Text & knowledge encoders are implemented using BERT. "Decoder" represents either the state tracker or the location predictor.
Text Encoder Given a paragraph P and an entity e, we first encode the text paragraph using a BERT model to obtain the contextual embeddings of each text token.
State Tracking Modules For each sentence X t , we concatenate the contextual embeddings of the entity h e t (if exists, otherwise masked as zero) and the verb h v t as input h s t to the state tracking modules. These modules include a knowledge injector, which infuses ConceptNet knowledge with h s t , and a Bi-LSTM state tracker that models the entity's state at each timestep t. Finally, a conditional random field (CRF) is applied to compute the conditional log likelihood of ground-truth state sequence y s and the state loss L state is computed as (1) where G denotes the knowledge graph extracted from ConceptNet, which will be elaborated in section 4.2.
Location Candidates For location prediction, we first extract location candidates {c 1 , · · · , c M } (possible location spans) from the paragraph. Specifically, we use an off-the-shelf POS tagger (Akbik et al., 2018) to extract all nouns and noun phrases as candidates, which can cover 87% of the ground-truth locations on the ProPara test set. We additionally define a learnable vector for location '?', which acts as a special candidate location.
Location Prediction Modules Similar to state tracking, for each location candidate c j at each timestep t, we concatenate the contextual embeddings of the entity h e t and the location candidate h c j,t as the input h l j,t to the location prediction modules. These modules also include a knowledge injector and a Bi-LSTM location predictor followed by a linear layer, which outputs a score o l j,t for each candidate c j at each timestep t. The scores of all location candidates at the same timestep are normalized using Softmax. Then the location loss L loc is computed as the negative log likelihood of the ground-truth locations: At inference time, we perform both sub-tasks, but only predict the entity's location when the model predicts its state as create or move.

Knowledge-Aware Reasoning
Here we explain the details of injecting ConceptNet knowledge into KOALA. We will use W and b to represent trainable weight and bias, respectively.

ConceptNet Knowledge Extraction
As a large relational knowledge base, ConceptNet is composed of numerous (h,r,t;w) triples, which means head concept h has relation r with tail concept t and w is its weight in the ConceptNet graph. For a given entity e, we first retrieve the entitycentric one-hop subgraph 1 from ConceptNet, i.e., entity e and its neighboring concepts. Then, we adopt two methods to retrieve relevant triples from this subgraph: • Exact-match: the neighboring concept appears in the paragraph P → {K e } • Fuzzy-match: the neighboring concept is semantically related to a content word in the paragraph P , according to contextual word embeddings → {K f }.
where {K e } and {K f } are sets of triples, sorted by weight w and semantic relevance 2 , respectively. We select the top N K triples so that |{K e }|+|{K f }| = N K , while priortizing exact-match ones. The detailed retrieval algorithm is presented in Algorithm 1. We set N K = 10 in practice. We manually evaluate 50 instances from the ProPara dataset. We divide the informativeness of the retrieved ConceptNet triples into 3 categories: providing direct evidence (36%), providing relevant knowledge (44%) and not relevant (20%). Among the first two categories, 75% of the instances can obtain new knowledge from the Con-ceptNet triples, which is not indicated in the text paragraph. This suggests the retrieved ConceptNet knowledge is very likely to be helpful from human perspectives.

Attentive Knowledge Infusion
The external knowledge is injected into our model in an attentive manner before the decoders 3 . We first encode the ConceptNet triples using BERT.

The BERT inputs are formatted as [CLS] head [SEP] relation [SEP] tail [SEP]
, where relation is interpreted as a natural language phrase. We use the average of BERT outputs (excluding special tokens) as the representation of a knowledge triple: In order to select the most relevant knowledge to the text paragraph, we use the decoder input as query to attend on the retrieved ConceptNet triples:

Algorithm 1 Knowledge retrieval on ConceptNet
Require: Entity-centric subgraph G composed of N G triples {τ 1 , · · · , τ N G }, Paragraph P composed of N P non-stopword tokens {w 1 , · · · , w N P }, entity e 1: K e ← ∅, K f ← ∅ 2: for τ i = (e, r i , n i ; w i ) in G do h τ i = BERT(p τ i ), h P = BERT(P ) 10: 11: where x ∈ {s, l} and g x t is the graph representation of the retrieved one-hop ConceptNet graph. Finally, we equip the decoder with an input gate to select information from the original input and the injected knowledge: i x find that such gated integration performs better than simply concatenating h x t and g x t together.

Attention Loss on Knowledge Infusion
Although the attention mechanism can help the model attend on knowledge relevant to the context, it is still challenging in some cases to find the most useful triple to the prediction target. In order to assist the model in learning the dependency between the prediction target and knowledge triples, we use an attention loss as explicit guidance. Specifically, we label a subset of knowledge triples that are relevant to the prediction target, and guide the model to attend more on these labeled triples.
A knowledge triple τ i is labeled as 1 ("relevant") at timestep t if: • y l t ∈ τ i and y s t ∈ {move, create}, which means the ground-truth location of the current movement/creation is mentioned in τ i .
• τ i ∩ V x = ∅ and y s t = x, where x ∈ {move, create, destroy}. V x is the set of verbs that frequently co-appear with state x, which is collected from the training set. This suggests that τ i includes a verb that usually indicates the occurrence of state change x.
The training objective is to minimize the attention loss, which is to maximize the attention weights of all "relevant" triples: where y r i,t ∈ {0, 1} is the relevance label of triple τ i at timestep t. Now the model is expected to better identify the relevance between textual input and ConceptNet knowledge during inference.
Finally, the overall loss function is computed as the weighted sum of three sub-tasks: 5 Multi-Stage Training

Multi-Stage Training on Wikipedia
As is mentioned in §1, we seek to collect additional procedural text documents from Wikipedia to remedy data insufficiency. is created by [SEP]
we adopt a multi-stage training schema for the text encoder in our model. Specifically, given the original pre-trained BERT model, we first perform selfsupervised language model fine-tuning (LM finetuning) on a procedural text corpus collected from Wikipedia, before performing further fine-tuning on the target ProPara or Recipes dataset. To collect additional procedural text, for each paragraph P in our target dataset, we split Wiki documents into paragraphs and use DrQA's TF-IDF ranker (Chen et al., 2017) to retrieve top 50 Wiki paragraphs that are most similar to P . Intuitively, we expand the training corpus by simulating the writing style of procedural text. Then, we fine-tune the vanilla BERT on these Wiki paragraphs using masked language model (MLM) objective.
In KOALA, contextual representations of entities, verbs and location candidates are used for downstream predictions. These tokens are mainly verbs and nouns. Therefore, in order to better adapt the fine-tuned BERT model to the target task, we only apply LM fine-tuning on nouns and verbs. In detail, each noun and verb receives a 0.3 mask probability 5 in MLM objective, whereas the other tokens are never masked in this phase. Thus, the fine-tuned BERT is able to generate better representations for nouns and verbs within procedural text corpora.

Multi-Stage Training on ConceptNet
Inspired by the above fine-tuning schema, we also adopt multi-stage training on the knowledge encoder, which is another BERT model. Different from the text encoder which encodes a sequence of unstructured text, the knowledge encoder models structured ConceptNet triples. Therefore, we modify the conventional MLM objective to fit the structural feature of ConceptNet triples.
Considering the bi-directional architecture of BERT, given a triple τ = (h,r,t), we iteratively mask out h, r and t (one at a time) and ask the encoder to predict the masked tokens using the other two unmasked components. If h or t consists of more than one tokens, we mask 50% of the tokens at a time to ensure trainability. However, we mask all tokens in r since the relation types in ConceptNet is limited (see Figure 3 for example). The encoder then learns to model the structural information of the knowledge triples through such LM fine-tuning. Similar to §5.1, the knowledge encoder is further fine-tuned while KOALA is trained on the target task.

Dataset
Our main experiments are conducted on the ProPara  dataset 6 . ProPara is composed of 1.9k instances (one entity per instance) out of 488 human-written paragraphs about scientific processes, which are densely annotated by crowd workers. As an auxiliary task, we also perform experiments on the Recipes (Bosselut et al., 2018) dataset 7 , which includes cooking recipes and their ingredients. In the original work, human annotation is only applied on the development and test set. Similar to (Gupta and Durrett, 2019a), we find that the noise in machine-annotated training data largely lowers models' performance. Therefore, we only use human-labeled data in our experiments and re-split it into 80%/10%/10% for train/dev/test sets. More statistics about these two datasets are shown in Table 1.

Implementation Details
For BERT encoders, we use the BERT BASE model implemented by HuggingFace's transformers library (Wolf et al., 2019). In LM fine-tuning, we set 6 https://allenai.org/data/propara 7 http://homes.cs.washington.edu/˜antoineb/ datasets/nyc_preprocessed.tar.gz batch size to 16 and learning rate to 5 × 10 −5 . The text encoder is trained for 5 epochs on Wikipedia paragraphs, while the knowledge encoder is trained for 1 epoch on ConceptNet triples. While finetuning the whole model on target dataset, we use batch size 32 and learning rate 3 × 10 −5 on Adam optimizer (Kingma and Ba, 2015). We set λ loc to 0.3 and λ attn to 0.5 in Eq.(12). Hidden size of LSTMs is set to 256 and the dropout rate is set to 0.4. We train our model for 20 epochs (~1 hour on a Tesla P40 GPU) and select the best checkpoint in prediction accuracy on the development set.

Evaluation Metrics
Doc-level task on ProPara 8 Document-level tasks, proposed by , require AI models to predict the input entities, output entities, entity conversions and entity moves in the procedural text. Evaluation metrics are average precision, recall and F1 scores on the above four objectives.
Sent-level task on ProPara 8 Sentence-level tasks, proposed by , require AI models to answer 3 sets of sentence-level questions: (Cat-1) Is entity e Created (Moved, Destroyed) in the process? (Cat-2) When is entity e Created (Moved, Destroyed)? (Cat-3) Where is entity e Created, (Moved from/to, Destroyed)? Evaluation metrics are macro-average and micro-average accuracy of three sets of questions.
Location change prediction on Recipes We evaluate our model on the Recipes dataset by how often the model correctly predicts the ingredients' location changes. We report precision, recall and F1 scores on this task.

Experiment Results
In our main experiments on ProPara (Table 2  and Table 3), we compare our model with previous works mentioned in §2. In the documentlevel task, KOALA achieves the best result on F1, which outscores the current state-of-the-art model DYNAPRO and our base model NCET by 7.5% and 12.6% respectively. In sentence-level tasks, KOALA outperforms previous models in most metrics, including macro-average and microaverage scores. These results show that KOALA 8 https://github.com/allenai/propara/tree/ master/propara/evaluation 9 https://leaderboard.allenai.org/propara/ submissions/public
In auxiliary experiments on Recipes, since we re-split the dataset using human-labeled data, we compare KOALA with its variants and our reimplemented NCET. As shown in Table 4, although not devised for cooking domain (e.g., retrieving ConceptNet triples using recipe ingredients may be noisy), our model still outperforms NCET and other variants in predicting location changes of recipes ingredients, which further proves the effectiveness of our model.

Ablations and Analyses
In order to further testify the effectiveness of the proposed components in this paper , we perform an ablation test on multiple variants of KOALA. As shown in Table 5, ConceptNet knowledge is proved to be effective even when we simply average their representations. Besides, both the attention mechanism and the attention loss contribute to selecting more useful knowledge from ConceptNet. As for multi-stage training, BERT encoders receive significant performance gain through fine-tuning on the ProPara task (66.5→68.7). The additional LM  Besides, we also compare the perplexity of the text encoder as an additional evaluation of multistage training. Here we use the nouns & verbs in the test set of ProPara as the evaluation target. As shown in Table 6, since ProPara contains many scientific terms which are usually nouns, vanilla BERT has a relatively high perplexity. However, LM fine-tuning on Wikipedia paragraphs largely reduces the perplexity by 64%. This indicates the fine-tuned text encoder performs better in predicting nouns & verbs, which leads to better token representations. This also show that the retrieved Wiki paragraphs successfully simulate the writing style of procedural text and covers the terminology of scientific processes. Considering results in Table  4-6, training with a larger corpus of procedural text indeed upgrades the model's performance.

Case Study
In Figure 4, we present two examples in ProPara test set where ConceptNet knowledge assists KOALA in making correct predictions. We list the predictions made with & without ConceptNet on the left, and visualize the attention weights assigned to ConceptNet triples while training with & without attention loss on the right. The first case shows how ConceptNet knowledge helps with more accurate state tracking. Although the paragraph does not explicitly state that the crater is created in sentence 3, ConceptNet knowledge tells the model that "crater can be formed from impacts", where "form" is a typical verb sign for action create. In fact, "form" is included in the co-appearance verb set V create that we collect from the training data. Although the vanilla attention finds some clues in knowledge triples, it also marks out irrelevant knowledge "crater is a type of geological basin", because V create has not been applied in training. After given the prompt of co-appearing verbs and trained with the attention loss, the model finally succeeds in paying major attention on the relevant knowledge triple.
In the second case, ConceptNet knowledge mainly helps predict the correct location for entity "cloud". Since the relationship between "cloud" and its location "air" is not mentioned in the paragraph, the model needs extra commonsense knowledge that clouds usually exist in the air. Fortunately, our model locates the relevant knowledge "cloud is at location air", while training with attention loss again emphasizes the importance of this knowledge piece. With the help of ConceptNet knowledge and the attention loss, our model is capable of collecting more information from both training data and external knowledge base, leading to more accurate predictions and better performance.

Conclusion and Future Work
In this work, we propose KOALA, a novel model for the task of procedural text understanding. KOALA solves two major challenges in this task, namely commonsense reasoning and data enrichment, using external knowledge sources. Extensive experiments on ProPara and Recipes datasets demonstrate the advantages of KOALA over various baselines. Further analyses prove that both ConceptNet knowledge injection and multi-stage training contribute to the strong performance of our model. Given the positive results achieved by KOALA, future work may focus on other issues on procedural text understanding, such as entity resolution or the implicit connection between verbs and states.