{'fraction_non_alphanumeric': 0.04358443488878271, 'fraction_numerical': 0.00966183574879227, 'mean_word_length': 4.2110650069156295, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Word-Label Alignment for Event Detection: A New Perspective via Optimal Transport

Event Detection (ED) aims to identify mentions/triggers of real world events in text. In the literature, this task is modeled as a sequence-labeling or word-prediction problem. In this work, we present a novel formulation in which ED is modeled as a word-label alignment task. In particular, given the words in a sentence and possible event types, the objective is to infer an alignment matrix in which event trigger words are aligned with the most likely event types. Moreover, we show that this new perspective facilitates the incorporation of word-label alignment biases to improve alignment matrix for ED. Novel alignment biases and Optimal Transport are introduced to solve our alignment problem for ED. We conduct experiments on a benchmark dataset to demonstrate the effectiveness of the proposed model for ED.


Introduction
Event Detection (ED) is one of the critical tasks in Information Extraction. Its goal is to identify and classify event triggers, i.e., the words/phrases that most clearly refer to the occurrence of an event of some predefined types in text. For example, in the sentence "Joe Biden was born on November 20, 1942", an ED system should recognize the word "born" as a trigger word of an event of type Birth.
A major challenge for ED is to assign an appropriate event type label for each word in a given sentence. In this work, we introduce a new perspective to solve ED as a word-label alignment problem that aims to align the set of words in the input sentence with the set of possible event type labels to represent correct label assignment for words. A key requirement for ED models in this new perspective involve inferring an alignment matrix to capture an alignment likelihood score between each pair of words and label types. The models can then be trained by enforcing the similarity between the predicted alignment matrix and the golden alignment matrix (computed from training data). In this way, previous ED models can be seen as a way to achieve the alignment matrix between words and labels where label distributions computed by the models serve as the alignment likelihood scores (Nguyen and Grishman, 2015;Chen et al., 2015;Wang et al., 2019;Cui et al., 2020;Ngo et al., 2021). However, given the word-label alignment perspective, previous ED models are suboptimal in at least two ways. First, the alignment likelihood scores in prior models are only used locally for each word (i.e., to compute the cross-entropy loss for each word to train models). The global uses of alignment matrix (e.g., to compute an overall distance between words and labels for training signals) are thus not yet explored in previous ED models. Second, current ED models mainly obtain alignment likelihood scores based on representation vectors for words and types, thus unable to exploit assignment biases to improve quality of the alignment matrix to train ED models. In particular, we propose two types of alignment biases that can be helpful for ED: (1) Word Preference: words with high likelihoods to be event triggers should be more aligned with event type labels (i.e., not the Other type for non-trigger words), and (2) Type Preference: event types that have higher chance to be appear in the input sentence should be associated with greater alignment scores. In all, we expect that global application and alignment biases can provide complementary information to boost current ED models in the new perspective.
To implement this idea, we propose to encode event trigger likelihoods for words and appearance likelihoods for event types as two distributions over words and event type labels (respectively) that will be induced from a deep learning architecture. Next, to inject the alignment biases into our ED model, we propose to feed the two distributions into Optimal Transport (OT) (Peyre and Cuturi, 2019) to induce an alignment matrix between words and event type labels. OT is an established framework 132 to find the optimal alignment between two distributions, thus providing a decent solution to incorporate alignment biases to compute alignment matrix in our ED problem. Finally, the induced alignment matrix will be leveraged to obtain a distance between words and event type labels, serving as a global application of the alignment matrix to introduce new training signals for ED. We conduct extensive experiments on a benchmark dataset to deliver state-of-the-art performance for ED. In summary, our contributions include: • A new perspective based on word-label alignment for event detection.
• Introduction of optimal transport to incorporate novel alignment biases for event detection.
• State-of-the-art performance for sequencelabeling event detection.

Model
Given an input sentence S = [w 1 , w 2 , . . . , w n ], the goal of ED is to predict the label sequence L = [l 1 , l 2 , . . . , l n ] where l i ∈ T is the label for the word w i ∈ S. Here, the label set T involves the BIO encoding tags for the event types in a given event ontology (e.g., B_Birth, I_Birth, and Other).
In this work, we propose to model ED as a wordlabel alignment problem where an alignment matrix is formed to capture the assignment likelihood for every pair of words in S and labels in T . We will first discuss word/label representations, and alignment matrix computation for training afterward.
Word & Label Representation: To represent the words in S, following prior work (Wang et al., 2019), we employ the pre-trained BERT model (Devlin et al., 2019). Concretely, the input sentence [[CLS], w 1 , w 2 , . . . , w n ] is fed into BERT to compute the contextualized embedding vectors E = [e cls , e 1 , e 2 , . . . , e n ]. We employ the average of vectors in the last layer of BERT to produce E. For the words with multiple word-pieces, we take the average of their word-piece representations.
To represent the event type labels l i , we employ a randomly initialized embedding table T in which every label is represented by a vector t i . The representations of the labels are updated during training. Alignment: To predict the label sequence L with our alignment idea, for every word w i , an alignment likelihood score a i,j between w i and each label l j is required (i.e., forming an alignment matrix A). Using the scores a i,j , the labell i can be predicted byl i = argmax j a i,j . Note that in prior ED models, the alignment scores a i,j are directly computed using the final task-specific feed-forward networks (Wang et al., 2019;Veyseh et al., 2021b). This approach is equivalent to computing the similarity between the representation vectors w i and t j , e.g., via dot-product. We call this approach "Vanilla Alignment". However, as discussed in the introduction, vanilla alignment scores a i,j are solely dependent on the learned representations e i and t j . As such, they cannot incorporate the alignment biases into the alignment matrix for ED.
To this end, we introduce two alignment biases that can be exploited to improve the word-label alignment for ED. In particular, for an effective ED model, we expect the words that are more likely to be event triggers to have higher alignment scores with event types. In contrast, the other words should be better aligned with the special label Other. i.e., non-trigger. We call this bias "Word Preference" for ED. In addition, among all event types, it is expected that the event types that have higher chance to be mentioned in the input sentence to be associated with greater scores in the alignment matrix A. We name this bias as "Type Preference". In this work, we aim to modify the vanilla alignment approach such that the two aforementioned preferences are observed. The quantification of Word and Type Preference and their incorporation into alignment matrix will be discussed in the following.
Word & Type Preference: To compute the word preference and type preference in the input sentence S, we consider two simpler versions of the ED problem. Specifically, for word preference, we utilize the Trigger Identification (TI) task that seeks to recognize the event trigger words without classifying them by event types. The event trigger probability computed for TI can be used to quantify the event trigger likelihood for each word w i ∈ S. Concretely, the representation e i of w i is fed into a feed-forward network with sigmoid activation function to compute the trigger likelihood p w i for w i : p w i = σ(F F w (e i )), where σ and F F w are sigmoid and feed-forward layer, respectively. To supervise the trigger likelihood scores, we include the binary cross-entropy loss function for TI into the overall loss for training: where y w i is a binary number to indicate whether if w i is a trigger in S. The likelihood scores p w i are employed to represent the word preference.
Next, for the type preference, we exploit the task of Type Prediction (TP) for ED. In this task, the objective is to predict which event types are mentioned in the sentence S (i.e., without predicting the trigger words). For an event type label t j , we predict the likelihood for t j to be mentioned in S by concatenating the type representation t j with the sentence representation e cls and feeding the result into a separate feed-forward network F F t with sigmoid activation to obtain the appearance like- ). To supervise the appearance likelihoods, the binary crossentropy loss function for TP is employed: , where y t j is a binary number to indicate the appearance of the event type t j in S. The likelihood scores p t j are utilized to represent the type preference. Alignment Computation: Given the word and type preference scores p w i and p t j , how can we compute an alignment matrix A between the words in S and the event type labels in T that can incorporate both word-label representation similarity (as in vanilla alignment) and designed preference scores for ED? Note that the preference scores can be modeled as two distributions over words and event type labels by applying a softmax function over the word and type likelihoods: D W P = sof tmax(p w 1 , p w 2 , . . . , p w n ) and D T P = sof tmax(p t 1 , p t 2 , . . . , p t T ). As such, we propose to employ Optimal Transport (OT) to elegantly combine the information to produce the alignment matrix A between S and T for ED.
Formally, given the probability distributions p(x) and q(y) over the domains X and Y, and the cost/distance function C(x, y) : X × Y → R + for mapping X to Y, OT finds the optimal joint alignment/distribution π * (x, y) with marginals p(x) and q(y) that converts p(x) to q(y) (i.e., the cheapest plan), by solving the following problem: Here, Π(x, y) involves all joint distributions with marginals p(x) and q(y). As such, the joint distribution π * (x, y) is a matrix whose entry (x, y) (x ∈ X , y ∈ Y) represents the probability of transforming x to y in the optimal transport. We use the Sinkhorn algorithm to approximately solve OT (Peyre and Cuturi, 2019). Finally, given π * (x, y), one approach to employ its global information is to compute the cost of optimal conversion Dist(π * ) = Σ x∈X Σ y∈Y π * (x, y)C(x, y) to measure the distance between X and Y (i.e., the Wasserstein distance).
To apply OT in our model, the domains X and Y are defined as the words w i ∈ S and types t j ∈ T ; the distributions p(x) and q(y) are set to the preference distributions D W P and D T P ; and the cost function C(w i , t j ) is computed using the Euclidean distance between the representations e i and t j . As such, solving the OT equation leads to the optimal alignment π * (w i , t j ), serving as our predicted alignment matrix (i.e., a i,j = π * (w i , t j )).
To train the ED model with word-label alignment, we propose two training signals obtained from the predicted alignment π * (e i , t j ). First, by treating the alignment score π * (e i , t j ) as the probability for w i to be assigned with label t j , we employ the negative log-likelihood loss to train our model: where l i is the golden label for w i in S. Second, we propose to globally enforce the similarity between the predicted alignment matrix π * (w i , t j ) from OT and the golden binary alignment matrix π g (w i , t j ) (i.e., π g (w i , t j ) = 1 if only if w i has the golden label t j ). As such, to aggregate the information in the alignment matrices, we first compute the Wasserstein distances Dist(π * ) and Dist(π g ) based on the predicted and golden alignments π * and π g . Afterward, we seek to minimize the difference between Dist(π * ) and Dist(π g ) to achieve alignment matrix similarity to train our ED models, leading to the loss: L OT = |Dist(π * ) − Dist(π g )|. Finally, the overall loss function for the entire model is L = α task L task +α OT L OT +α T I L T I +α T P L T P .

Experiments
Datasets & Baselines: We evaluate the performance of the proposed model (called OTED) on the ACE 2005 dataset (Walker et al., 2006) that annotates 599 documents for 33 event types in English. We use the same data split and preprocessing as prior work (Wang et al., 2019;Veyseh et al., 2021b) for this dataset. The numbers of documents for the training/development/test data are 529/30/40 respectively. Following (Wang et al., 2020a;Veyseh et al., 2021b) triggers to span multiple words.
As the baselines, we compare with the typical sequence labeling models for ED, i.e., BiLSTM, DM-BERT (BERT with dynamic multi-pooling), and BERT+CRF in (Wang et al., 2020a), and the prior state-of-the-art (SOTA) model reported for ACE 2005, i.e., ED3C (Veyseh et al., 2021b). For all the models, we use the same version of pre-trained BERT base to achieve a fair comparison. Following prior work (Wang et al., 2020b;Veyseh et al., 2021b), we use span-based precision, recall and F1 scores for correctly predicting the boundaries and types of event triggers as the performance metrics. Finally, we fine-tune the hyper-parameters for OTED using the development data of ACE 2005. In our model we use the BERT base model to encode data; 2 layers for all the feed-forward neural networks with 200 hidden dimensions in the layers. The trade-off parameters α task , α OT , α T I and α T P are set to 1.0, 0.01, 0.05, and 0.01 respectively. The learning rate is set to 3e-5 for the Adam optimizer and the batch size of 8 is employed during training. Results: The model performance is presented in Table 1. This table shows that OTED significantly outperforms the baseline models on ACE 2005. We attribute the superiority of OTED to its capability to incorporate alignment biases, i.e., word and type preference, into alignment-based ED. The better performance of OTED over ED3C is important as unlike this baseline OTED does not require additional document context or supervision from other related tasks. Ablation Study: We conduct an ablation study for the components of OTED over the ACE 2005 development set. Table 2 presents the performance of three groups of ablated models for OTED. In the first group (lines 2-4), we exclude one or both alignment biases, i.e., WP and TP, from OTED. Concretely, to remove a preference, its corresponding distribution in the OT (i.e., D W P and D T P )  is replaced with the uniform distribution in the OT computation for OTED. It is clear from the table that both alignment biases are beneficial for OTED as removing any of them would hurt the performance significantly. Next, the second group (lines 5-8), we exclude each loss component (i.e., L task , L OT , L T P , and L T I ) from the overall loss L to train OTED. As can be seen, all the designed losses contribute significantly to the performance of OTED, thus testifying to their effectiveness in alignment-based ED. Also, in the third group (lines 9-10), we explore two variants of OTED to justify the design of the loss L OT to incorporate OT into the model. In one variant (called Distance in line 9), instead of minimizing the difference L OT between the Wasserstein distances based on predicted and golden alignments, we directly minimize the predicted Wasserstein distance Dist(π * ) between words and labels. Moreover, in the Alignment variant in line 10, instead of employing the Wasserstein distance, we directly minimize the distance between the predicted and golden alignment π * (w i , t j ) and π g (w i , t j ) (i.e., evaluated by i,j |π * (w i , t j ) − π g (w i , t j )|/(n|T |)). As can be seen, both Distance and Alignment lead to inferior performance for OTED, thereby showing the effectiveness of L OT for ED. As such, we attribute the poor performance of Distance to the lack of supervision from the golden alignment-based distance π g (w i , t j ), and the worse performance of Alignment to the missing of contextual similarity (i.e., the cost C(w i , t j )) in the distance computation. Analysis: In this section, we present a qualitative analysis to shed more light on the superiority of the proposed model OTED to the prior sequence labeling methods. Specifically, we compare our model with the BERT+CRF baseline by analyzing the examples in which BERT+CRF fails to recog-  nize the event types and triggers, but OTED can successfully perform the predictions. A major findings in our analysis is that OTED can exploit the introduced alignment bias (i.e., word and type preference) to avoid unlikely event triggers and types (i..e, the ones that should be obviously eliminated based on overall sentence context). This leads to correct predictions for examples that BERT+CRF make mistakes. Table 3 shows two examples from the development set of the ACE 2005 dataset to illustrate our findings. In the first example, the baseline can recognize the event trigger "shot", but fails to predict the event type. Given the context of the sentence, the predicted event type Contact:Meet by BERT+CRF should be considered as unlikely to be mentioned in the sentence. As the proposed model OTED employs type preference knowledge, it successfully avoids unlikely event types for this sentence. In addition, in the second example, the baseline incorrectly predicts a non-trigger word (i.e., "waiting") as a trigger. In contrast, since OTED employs word preference knowledge, it can effectively avoid unlikely event triggers.

Conclusion
We present a general word-label alignment formulation for ED in which each pair of words and types is associated with an alignment score for label assignment likelihood. Moreover, we introduce two alignment biases based on type and word preference to improve the word-label alignment matrix computation with OT. Extensive analysis on a benchmark dataset demonstrates the benefits of the proposed technique for ED. In the future, we plan to evaluate our method on more datasets for ED (Wang et al., 2020a;Man et al., 2020; to better understand its operation.