{'fraction_non_alphanumeric': 0.037785903778590375, 'fraction_numerical': 0.016096651609665162, 'mean_word_length': 4.897463002114165, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Simple and Unified Tagging Model with Priming for Relational Structure Predictions

Relational structure extraction covers a wide range of tasks and plays an important role in natural language processing. Recently, many approaches tend to design sophisticated graphical models to capture the complex relations between objects that are described in a sentence. In this work, we demonstrate that simple tagging models can surprisingly achieve competitive performances with a small trick -- priming. Tagging models with priming append information about the operated objects to the input sequence of pretrained language model. Making use of the contextualized nature of pretrained language model, the priming approach help the contextualized representation of the sentence better embed the information about the operated objects, hence, becomes more suitable for addressing relational structure extraction. We conduct extensive experiments on three different tasks that span ten datasets across five different languages, and show that our model is a general and effective model, despite its simplicity. We further carry out comprehensive analysis to understand our model and propose an efficient approximation to our method, which can perform almost the same performance but with faster inference speed.


Introduction
Many tasks in natural language processing require extracting relational structures from the given context. For example, the event argument extraction task aims to identify event arguments and their corresponding roles to the given event trigger (Huang et al., 2022;Wang et al., 2019). In end-to-end relation extraction, the model recognizes the relationship(s) between tail-entities and head-entities Yu et al., 2020). In task-oriented * The authors contribute equally. semantic parsing, the model predicts slots and semantic roles of the intent (Tür et al., 2010;. These tasks are beneficial for a wide range of applications, such as dialog system , building knowledge graph , question answering (Berant et al., 2014;Yasunaga et al., 2021), and narrative generation (Chen et al., 2019a).
Various task-specific methods have been proposed for different tasks that demand relational structure extraction. Graph-based methods are one of the most commonly used frameworks to deal with such problems Miwa and Bansal, 2016;Fu et al., 2019). These methods make relational extraction by considering the interaction between the objects (such as event triggers, named entities, etc.) and their relationships. By leveraging the relational interaction, they iteratively update the representation for each object to make its representation more comprehensive and better suit for modeling the whole graph structure, as illustrated in Figure 1(a).
On the other hand, sequential tagging models are conventional methods for addressing the extraction of relational structures (Yang et al., 2019a;Hakkani-Tür et al., 2016;Ma et al., 2020;Zhang et al., 2018;Ramponi et al., 2020). These methods convert the relational structure into a sequential format and make predictions by labeling tokens in the input passage. Compared to graph-based models, tagging models get less attention in recent studies since graph-based methods are expected to capture more complex interaction between objects and thus lead to stronger results.
In this work, we re-examine the effectiveness of sequential tagging methods on relational structure extraction and present a simple and unified, yet   Fincke et al. (2022), we introduce sequential tagging model with priming, which augments the input with the task-specific context and perform sequential labeling on the input passage, as shown in Figure 1(c)&(d). The priming approach that makes the sentential representations task-specific well employs the contextualized pre-trained language models' nature. Hence, compared to the commonly used feature embedding approach, it better exploits the capability of sequential tagging methods for relational structure extraction, as we will show in Section 4.
To better understand the mechanism of priming, we carry out experiments to analyze whether deeply-stacked layers is essential for priming method to equip the sentential representation with task-specific information. We observe that although deeper layers help, priming sequential models with much shallower layers can achieve similar performance. Based on the study, we further introduce an efficient approximation of the sequential tagging model with priming and the approximating model can achieve similar performance but faster inference speed.
Our contributions can be summarized as follows. First, we present a simple and unified sequential tagging model that can tackle several relational structure extraction tasks and serve as a strong baseline for further studies. The model is verified in three different tasks and in various languages. The results are comparable or achieve a new stateof-the-art on three different natural language processing tasks in ten datasets in total. Second, we perform a thorough analysis on the priming methods, and propose a novel efficient approximation to fasten up models during inference time with only a small performance drop.

Relational Structure Extraction with Sequential Tagging
In this section, we first introduce a unified formulation of relational structure extraction (Section 2.1) and how previous works usually apply sequential tagging model to relational structure extraction (Section 2.2). Then, we present the improved sequential tagging model with the priming technique (Section 2.3).

A Unified Formulation of Relational Structure Extraction
Given a passage x = [x 1 , x 2 , ..., x n ] and a condition c, a relational structure extraction model identifies a list of objects and their corresponding relationships r c = [r c 1 , r c 2 , ..., r c l ] to the condition c, where r c i ∈ S r and S r is the set of all possible relationships. Many relational structure extraction tasks can be approached by this formulation and applied to several natural language processing (NLP) tasks. In this paper, we specifically focus on the following three NLP tasks.
End-to-end event extraction. End-to-end event extraction aims to extract events, each of which consists of an event trigger and several arguments with their specific roles, from a given passage (Ma et al., 2020;Hsu et al., 2022b;Yang et al., 2019a). We consider a pipelined solution such that, after the event triggers are identified, a relational structure extraction model extracts the event arguments and their corresponding roles for each given event trigger. In this formulation, the condition c is the given event trigger, and the objects o c and the relationships r c are the event arguments and their argument roles, respectively.
End-to-end relation extraction. End-to-end relation extraction extracts entities and their relations from text. We consider the cascading pipelined approach to solve the task Yu et al., 2020;. After named entities are extracted, we apply a relational structure extraction model to predict tail-entities and their relationships for each extracted named entity that serve as the "head-entity". Thus, each given head-entity is the condition c, the extracted tail-entities are o c , and their relationships are r c .
Task-oriented semantic parsing. Task-oriented semantic parsing intends to classify the intent and parse the slots in an utterance (to a task-oriented dialog system) Gupta et al., 2018). In our framework, we first predict the intent and use a relational structure extraction model to predicts the slots (o c ) and the semantic roles (r c ) in the utterance for the given intent, which serves as the condition c.

Sequential Tagging Model for Relational Structure Extraction
We introduce the typical way to apply a sequential tagging model to solve relational structure extraction. The goal of a sequential tagging model for relational structure extraction is to predict BIO-tags (y i ) for each token (x i ) in the input passage. The BIO-tag sequence can then be decoded to represent the objects o c (and their relation r c ) to be extracted. Specifically, given an input passage, we obtain the contextualized representation z i for each token x i by passing the passage to a pre-trained language model. 1 To embed the information of the condition c, one commonly-used technique is to add conditional features to z i (Ma et al., 2020;Yang et al., 2019a;Yu et al., 2020). Following Ma et al. (2020), we consider a condition token feature and a learnable condition type feature, as shown in Figure 1(b). The token feature of c is the contextualized word representation z j , if x j is the token that represents the condition. The learnable type feature helps the modeling of the attribute of a condition, such as the condition's event type in event argument extraction. Augmented with these conditional features, the final representation for token x i is further fed into multi-layer perceptrons and a conditional random field layer (Lafferty et al., 2001) to predict each token's corresponding tag y i .

Sequential Tagging Model with Priming
Condition Priming. Motivated by previous work (Fincke et al., 2022), we consider priming to inject the information of the condition c to further improve the sequential tagging model. The priming mechanism informs the model the conditional information by directly appending conditional information to the input of pretrained language models. However, unlike Fincke et al. (2022) that uses an integer string to represent features in a categorical style, we use a natural-language-styled indicator to better exploit the semantics of the condition. The indicators can be obtained by simply verbalizing the conditional information.
Take Figure 1(c) as an example, when extracting the tail-entities and the relationships for the "military" head-entity, we first verbalize the entity type Model ACE05-E (en) ACE05-E (zh) ERE (en) ERE (es) Tri-C Arg-I Arg-C Tri-C Arg-I Arg-C Tri-C Arg-I Arg-C Tri-C Arg-I Arg-C  Table 1: Results of end-to-end event extraction. All values are micro F1-score, and we highlight highest scores with boldface. * We reproduce the results using their released code.
of "military", i.e, from "Org" to "Organization". Then, the string "Organization" will be concatenated to the input sequence together with the token "military", which serves as the token information about the condition c.
Our priming technique leverages the contextualized pre-trained language models' nature to make the token representation z i condition-aware. Hence, the representation of every z i is more task-specific than the one in the model that described in Section 2.2. More precisely, for tagging models without priming, z i usually captures a more general representation that focuses on the context of input passage. For models with priming, the contextualized representation z i is affected by the additional verbalized words inherently, hence, becomes task-specific and more suitable for addressing the task (Zheng and Lapata, 2022;Zhong and Chen, 2021). Additionally, the priming method can be easily combined with learnable features, and the discussion on the learnable feature versus priming method will be shown in Section 4.
Relation Priming. The same idea can also be extended to relation. We first decompose a relational structure extraction task into several extraction subtasks, each of that only focuses on relational extraction for one single relation r (r ∈ S r ). Under this setting, the tagging model's input contains a verbalized relation string for the given relation r. Thus, z i is aware of the queried relation r and suitable for predicting whether any object in the passage forms a relationship with the c in the type r.
For example, in Figure 1(d), for the given relation "Part-Whole", we first verbalized it into "is part of". Then, the "is part of" string is appended to the input sequence for the model. The BIOtagging sequence can be decoded into those tailentities o c that form "Part-Whole" relationship(s) with the head-entity "military".

Experiments
To study the effectiveness of the sequential tagging method, we conduct experiments on end-toend event extraction, end-to-end relation extraction, and task-oriented semantic parsing. All the experimental results are the average of five runs with different random seeds.

End-to-End Event Extraction
Datasets. We consider English and Chinese annotations in ACE-2005 (ACE05-E) (Doddington et al., 2004) and keep 33 event types and 22 roles, as suggested in previous works . We also consider English annotations and Spanish annotations in ERE (Song et al., 2015) and follow the preprocessing of  to keep 38 event types and 21 roles.
Baselines. We consider the following end-to-end event extraction models, including DyGIE++ (Wadden et al., 2019), TANL (Paolini et al., 2021), Text2Event (Lu et al., 2021), OneIE , and DEGREE (Hsu et al., 2022b). Since TAGPRIME requires trigger predictions, we simply use the trigger predictions made by a sequential tagging model trained with multi-tasking on trigger detection and named entity recognition.
For TAGPRIME, DyGIE++, and OneIE, we consider BERT-large for ACE05-E (en) and ERE (en), and consider XLM-R-large for ACE05-E (zh) and ERE (es). For generation-based models, we consider BART-large for DEGREE, T5-base for TANL, and T5-large for Text2Event, as their original papers suggest.
Evaluation metrics. Following previous works , we measure the correctness of arguments based on whether the offsets of the argument spans match or not. We consider argument identification F1-score (Arg-I), which cares about only the offset correctness, and Model ACE05-R ACE04-R Ent Rel Rel+ Ent Rel Rel+  argument classification F1-score (Arg-C), which cares about both offsets as well as the role types. We also report trigger classification F1-score, although this is not our main focus.
Results. Table 1 shows the results of end-to-end event extraction on various datasets and languages. Although being simple, TAGPRIME surprisingly has a decent performance and achieves better results than the state-of-the-art models in terms of argument F1-scores. We attribute the good performance to the design of priming, which leverages the semantics of the condition and makes the representations more task-specific. Considering relation priming further improves the results, which again shows the importance of task-specific representations.

End-to-End Relation Extraction
Datasets. We test on two popular used endto-end relation extraction datasets ACE04 and ACE05 (Doddington et al., 2004), denoted as ACE04-R, and ACE05-R. Various domains are covered in both datasets, such as newsire and online forms, and both datasets consider 7 name entity types and 6 different relations. We follow the same procedure in Zhong and Chen (2021) to preprocess and split the datasets. We refer readers to the paper for more dataset details.
Baselines. We compare to the following end-toend relation extraction models: Table-Sequence (Wang and Lu, 2020), Cascade-SRN (Late fusion) (Wang et al., 2022), and Cascade-SRN (Early fusion) (Wang et al., 2022). Additionally, we consider PURE (Zhong and Chen, 2021), which also takes a pipelined approach to solve end-to-end relation extraction, i.e, first train an named entity recognition models and predict reltionship(s) among predicted named entities. To fairly compare with prior works, we use the PURE's named entity predictions on the test set and feed them to TAGPRIME to perform relational structure extraction. 2 For all the models, we consider the single sentence setting and use ALBERT-xxlarge-v1 as the pretrained language model.
Evaluation metrics. We follow the standard evaluation setting with prior works (Bekoulis et al., 2018;Zhong and Chen, 2021) and use micro F1 measure as the evaluation metric. For named entity recognition, a predicted entity is considered as a correct prediction if its span and the entity type are both correct. We denote the score as "Ent" and report the scores even though it is not our main focus for evaluation. For relation extraction, two evaluation metrics are considered: (1) Rel: a predicted relation is considered as correct when the boundaries of head-entity span and tail-entity span are correct and the predicted relation type is correct; (2) Rel+: a stricter evaluation of Rel, where they additionally required that the entity types of head-entity span and tail-entity must also be correct.
Results. The results of end-to-end relation extraction is presented in Table 2. From the table, we observe that TAGPRIME has the best performance on ACE05-R and outperforms most baselines on ACE04-R. This shows the effectiveness of TAGPRIME. Similar to the results of event extraction, considering relation priming makes the representations more relation-aware and leads to the performance improvement.

Task-Oriented Semantic Parsing
Datasets. We choose MTOP , a multilingual dataset on semantic parsing for taskoriented dialog systems. We specifically consider data from English (en), Spanish (es), French (fr), and German (de).

MTOP (en) MTOP (es) MTOP (fr) MTOP (de) Intent Slot-I Slot-C Intent Slot-I Slot-C Intent Slot-I Slot-C Intent Slot-I Slot-C
JointBERT  96.   Baselines. We consider JointBERT (Chen et al., 2019b), the commonly used baseline for taskoriented semantic parsing. We directly use the predicted intents by JointBERT as the condition of TAGPRIME for fair comparison. Both TAGPRIME and JointBERT are trained with XLM-R-large. Unlike event extraction and relation extraction, the condition of task-oriented semantics parsing (intent) does not include the word span, we therefore remove the condition token feature for TAGPRIME in this experiment.
Evaluation metrics. Following MTOP , we consider slot identification F1-score (Slot-I) and slot classification F1-score (Slot-C). Despite we focus on the performance of slot filling, we also report the intent classification accuracy. Table 3, TAGPRIME achieves a better performance than the baselines. Again, considering relation priming leads to further improvement. It is worth noting that TAGPRIME is effective for different languages, which shows the generality of TAGPRIME.

Summary
We show the superiority of TAGPRIME on three different tasks (including ten different datasets that across five different languages). The results suggest that TAGPRIME is a simple but effective model for general relational strucutre extraction tasks.

Analysis
In this section, we study two questions: (1) What is the effectiveness of priming techniques compared to learnable features? (2) Whether the deeplystacked layers to perform priming is essential?
To answer the first question, we conduct ablation experiments on sequential tagging models using different combinations of learnable feature or/and adding information through the priming mechanism (Section 4.1). For the second question, we propose a simple modification to TAGPRIME so that we can flexibly control the number of layers to fuse priming information to token representations. Moreover, we show that the modified TAGPRIME can serve as an efficient approximation of TAG-PRIME (Section 4.2).

Ablation Study
To conduct the ablation studies on sequential tagging models, we focus on the setting that we alter the choice on how to include the type information of the condition c and the relation information. Table 4 demonstrates our experimental results.
Comparing the first four cases in Table 4, we can observe that the addition of type features is useful in general, and using our priming mechanism is a more effective way to incorporate conditional information. For models in case 5 to case 8 in Table 4, the decomposition formulation described in Section 2.3 is applied. Comparing case 2 to case 5, we can see that simply applying the relation decompo-sition formulation for solving relational structure extraction does not lead to major improvements if the way to embed the queried relation is only through learnable features. However, comparing case 3 to case 6 and case 4 to case 7, we show that the relation priming approach make the representation z i well capture the attribute of the queried relation, thus, better exploit the advantage of the relation decomposition formulation and gain larger improvements. It is worth noting that we conduct a preliminary experiments that use pretrained language models' representations of the same verbalized token to be the initialization of the learnable type feature embedding, but the method show similar results with random initialization, hence, we stick to random initialization on the models with learnable feature embeddings.

Depth of Layers to Prime Models
The difference between sequential tagging model with only learnable features and TAGPRIME is whether the contextualized representation z i is affected by additional verbalized words or not. The deeply-stacked pretrained language model performs several layers of self-attention to gradually embed the verbalized information into z i . We want to study whether the deeply-stacked layers to influence z i is essential. We perform studies on TAG-PRIME w/ Condition & Relation Priming model.
We now perform two modifications to TAG-PRIME: (1) we first separate the pretrained language model, which contains L layers, into two halves -one with the first k layers, the other one is the remaining layers, and (2) we copy the first half to be another module. When an input passage is fed into the model. We use the original first half to encode the passage and verbalized condition, and use the copied first half to encode the verbalized relation. Finally, the encoded representations will be fed into the second half layers. With this model design, only (L − k) layers is adapted for the verbalized information to influence the contextualized representation z i , as illustrated in Figure 2. We alter the value of k, where when k = 0, it represents the TAGPRIME w/ Condition & Relation Priming, and when k = L, it is TAGPRIME w/ Condition Priming model.
We conduct experiments on the ACE05-E (en) dataset (an event extraction dataset). In order to better analyze the results and isolate the influence from pipelined error, we report the results on the event argument extraction, which assumes gold event triggers are given. The performance are shown in Figure 3a. From the figure, we find that when k ≤ 10, the performance our modified TAGPRIME is strong in general and is comparable with TAG-PRIME w/ Condition & Relation Priming. Hence, we conclude the answer toward our second question to -as long as the layers to perform priming is deep to an extent, the benefits of priming mechanism can be anticipated.
One additional benefit of our modified TAG-PRIME is its efficiency during inference. Since the encoding stage of the passage and the verbalized relation is separated, we can fasten up the inference time of our modified TAGPRIME via parallel encoding. More precisely, our modified TAGPRIME can aggregate instances that share the same passage and verbalized condition. For those instances, TAGPRIME only needs to perform the encoding for one time on their major part, 3 and paired with several separated embedded verbalized relation, which could be parallelly encoded together.
To compare the efficiency of the model, we benchmark the inference time through performing inference on the whole testing dataset fifty times and calculate the average speed, which is measured by checking how many instances can be processed per second. The results are in Figure 3b. From Figure 3, we observe that the for our modified TAG-PRIME with k = 10, its inference speed is around 30% faster than the TAGPRIME w/ Condition & Relation Priming, but they perform similarly.

Related Work
Relational structure extraction has been studied for over a decade (Riedel, 2009) together with its wide range of applications in natural language processing tasks, such as event extraction, relation extraction, task-oriented semantic parsing, semantic role labelling, coreference resolution, etc. The prevalence of its application makes us hard to exhaust them in this short summary, hence, in this section, we mainly focus on related works that experiments with the same tasks with us.
Event extraction. Early works in event extraction mostly consider a pipelined approach (Nguyen and Grishman, 2015; Wang et al., 2019;Yang et al., 2019b;Huang et al., 2022;Du and Cardie, 2020) to deal with event extraction. Some followup works argue that pipelined design leads to error propagation issues and hence propose end-to-end approaches to better capture dependencies between each predictions Li et al., 2013;Yang and Mitchell, 2016;Nguyen et al., 2016;Hsu et al., 2022b;Lu et al., 2021;Huang and Peng, 2021). However, recently some empirical studies (Hsu et al., 2022b;Zhong and Chen, 2021;Fincke et al., 2022) also show that given abundant amount of data and properly learned distinct contextualized representation for each pipelined tasks, it is hard to conclude that jointly learned approaches always provide a stronger result. This aligns with our discovery and experimental results -even though we apply a pipelined approach with simple sequential tagging framework on event extraction, with the help of priming to learn more condition-aware contextualized representation, we can still achieve very strong performance on multiple datasets in various languages.
Relation extraction. Ways to solve end-to-end relation extraction can be briefly classify into three categories. The first one is directly perform joint inference on named entities and their relationship(s) (Zheng et al., 2017;Wang and Lu, 2020;Katiyar and Cardie, 2017;Miwa and Bansal, 2016;Fu et al., 2019). Anther is to perform a pipeline that first extract named entities first, and then perform relation classification (Wu and He, 2019;Hsu et al., 2022a;Lyu and Chen, 2021;Peng et al., 2020;Zhou and Chen, 2021;Lu et al., 2022), which assume both the head-entity and tailentity are given. The other is the cascading approaches that we use in the paper Yu et al., 2020;. Among them, many sophisticated graph-based methods emerge rapidly recently Miwa and Bansal, 2016;Fu et al., 2019). In contrast, sequence labelling based approach (Miwa and Bansal, 2016;Zhang et al., 2017) are getting less attention. This highlights one of our motivation to re-examine the merit of sequence tagging, the simple and concise approach, and we show that with our simple modification to the model, this classic framework can serve as a strong baseline.
Task-oriented semantic parsing. Most works on task-oriented semantic parsing focus on intent classification and slot filling tasks (Tür et al., 2010;Gupta et al., 2018;Hakkani-Tür et al., 2016;Zhang et al., 2018;Louvan and Magnini, 2020). Recently, some more advanced neural network based approaches have been proposed, such as MLP-mixer (Fusco et al., 2022) or sequence-to-sequence formulation (Desai et al., 2021). Yet, JointBERT (Chen et al., 2019b) a sequence-tagging based model that is jointly trained model to classify intent and extract slots, still serve as a widely-used baseline due to its simplicity. Our approach enjoys the same benefit on simplicity with the JointBERT and can further enhanced its performance.

Conclusion
In this paper, we present TAGPRIME, a simple but effective and general sequential tagging model.
The key idea is applying priming, a small trick to make the representations condition-aware by appending condition-related string to the input passage. Our experimental results demonstrate that TAGPRIME has consistent improvements over different relational structure extraction tasks and is general to various languages. Finally, we show that TAGPRIME can be accelerated without much performance loss by reducing the encoding layers.