{'fraction_non_alphanumeric': 0.034277387062028764, 'fraction_numerical': 0.012925299372742824, 'mean_word_length': 4.832963784183296, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 1, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
TREND: Trigger-Enhanced Relation-Extraction Network for Dialogues

The goal of dialogue relation extraction (DRE) is to identify the relation between two entities in a given dialogue. During conversations, speakers may expose their relations to certain entities by explicit or implicit clues, such evidences called"triggers". However, trigger annotations may not be always available for the target data, so it is challenging to leverage such information for enhancing the performance. Therefore, this paper proposes to learn how to identify triggers from the data with trigger annotations and then transfers the trigger-finding capability to other datasets for better performance. The experiments show that the proposed approach is capable of improving relation extraction performance of unseen relations and also demonstrate the transferability of our proposed trigger-finding model across different domains and datasets.


Introduction
The goal of relation extraction (RE) is to identify the semantic relation type between two mentioned entities from a given text piece, which is one of basic and important natural language understanding (NLU) problems (Zhang et al., 2017;Zhou and Chen, 2021;Cohen et al., 2020). In this task setting, we are usually given a written sentence and a query pair containing two entities and asked to return the most possible relation type from a predefined set of relations. Dialogue relation extraction (DRE), on the other hand, aims to excavate underlying crosssentence relation in natural human communications (Yu et al., 2020;Jia et al., 2021). The problem itself is well-motivated, because relations between entities in dialogues could potentially provide dialogue systems with additional features for better dialogue managing (Peng et al., 2018;Su et al., 2018a) or response generation (Su et al., 2018b). 1 The source code is available at: http://github. com/MiuLab/TREND.  There are two popular datasets, DialogRE (Yu et al., 2020) and DDRel (Jia et al., 2021), focusing on relation extraction in dialogues illustrated in Figure 1. In DRE, given a conversation and a query pair, we aim to identify the interpersonal relationship between the given entities, where entities can be human or other types like locations. As shown in Figure 1, the evidences of relations within the conversation flow, called Triggers, provide informative cues for this task. A trigger can be a short phrase or even a single word with any possible part-of-speech. In the example, the clue for knowing the speaker 2 has a negative impression on the speaker 3 comes from the sentence "You are arrogant." Such hint is intuitively useful for deciding the relations. However, Albalak et al. (2022) is the only prior work that tried to explicitly leverage such signal for improving DRE, because such explanation annotations may not be always available (Kung et al., 2020).
Prior work can be divided into two main lines, one of which is graph-based methods. DHGAT (Chen et al., 2020) presents an attention-based heterogeneous graph network to model multiple types of features; GDPNet (Xue et al., 2021) constructs latent multi-view graphs to model possible relationships among tokens in a long sequence, and  then refines the graphs by iterative graph convolution and pooling techniques. Another branch is BERT-based (Kenton and Toutanova, 2019) methods (Yu et al., 2020;Xue et al., 2022). SimepleRE (Xue et al., 2022) is a simple BERT model with an additional refinement gate for iteratively finding high-confidence prediction. LSR (Nan et al., 2020) is a latent structure refinement method for better reasoning in the document-level relation extraction task. Although it is known that using trigger information can significantly help the performance of relation extraction, only DialogRE has the annotated triggers. It is not guaranteed that utilizing the annotated triggers can generalize to other relations from other datasets, considering the discrepancy of their relation types.
Given the target data without trigger annotations, this paper proposes TREND, a simple multitasking model with an attentional relation predictor, where it learns the general capability of finding triggers and transfers it to the unseen relations for performance improvement. The experiments show that our proposed method can effectively identify the explicit triggers and generalize to unseen relations towards great flexibility and practicality.

Proposed Method
The core idea of this model is to identify trigger spans and accordingly leverage such signal to improve relation extraction. We hereby propose Trigger-enhanced Relation-Extraction Network for Dialogues, TREND, illustrated in Figure 2.

Problem Formulation
Given a piece of dialogue context D composed of text tokens D = {x i } and a query pair q containing a subject entity and an object entity q = (s, o), the task aims at learning a function f that finds the most possible relations between the given entities from a predefined relation set R, f (D, q) → R. Note that a single query pair may contain multiple relations, and we duplicate the data samples when they have multiple relation labels by following the prior work.

TREND
The proposed model has two modules, (1) a multitasking BERT (Kenton and Toutanova, 2019) for encoding context and identifying triggers, and (2) a relation predictor with a feature fusion of the dialogue and the automatically identified trigger.
As illustrated in Figure 2, an input (D, q) will be first augmented into a BERT-style sequence.
Specifically, the input format is We replace the target entity pair with their speaker tokens in D following Yu et al. (2020) illustrated in the figure. The first [CLS] encodes the dialogue contexts, and the second one is to predict whether the triggers are explicit via binary classification detailed below.
Explicit Trigger Gate Because triggers sometimes are implicit, it is difficult to identify the associated trigger spans of dialogue relations. We hereby propose to learn a binary classifier as a gate to identify if the explicit triggers exist, and empty trigger spans are inputted to relation prediction when no explicit triggers. The binary cross entropy loss L binary is used here.

Trigger Prediction
The explicit triggers are identified by a extractive method with start-end pointer prediction (Kenton and Toutanova, 2019), which is prevalent in extractive question answering (Lee et al., 2016;Rajpurkar et al., 2016). This is a single-label classification problem of predicting the most possible positions; hence a cross entropy loss L trigger is conducted.
Relation Prediction A learned context vector and a predicted trigger span are then fed into the relation predictor as depicted in the top part of Figure 2. The features are fused by a generic attention mechanism, where the query is the context vector c, and the keys and the values are trigger words x i encoded by BERT: softmax(c·x i )·x i . The merged feature is then fed into a 1-layer feedforward network for final relation prediction using a cross entropy loss L relation .
Supervised Joint Learning Considering that only DialogRE contains the annotated trigger cues, we perform supervised joint learning for three above tasks. Three above losses are linearly combined as the learning objective for training the whole model in an end-to-end manner. The weights for adjusting the impact of each loss are tuned in the development set. We also apply schedule sampling (Bengio et al., 2015) on explicit trigger classification and trigger prediction when feeding into the relation predictor in order to mitigate the gap between the true triggers and the predicted ones.
Transfer Learning Because annotated triggers may not be available, this paper focuses on transferring the trigger-finding capability to another target dataset, DDRel, which does not contain trigger annotations and its relation types differ a lot from DialogRE. We replace the final feed-forward layer with a new one, since relation numbers may different in two datasets. Then we fine-tune the whole model using a single loss about relation prediction, L relation , where we assume the trigger-finding capability can be better transferred cross different datasets/relations.

Experiments
We focus on evaluating the performance of DRE on the dataset without trigger labels in order to investigate if the trigger-finding capability can be transferred across datasets/relations.

Setting
The DRE datasets used in our experiments are Di-alogRE (v2) with trigger annotations (Yu et al., 2020) and DDRel (Jia et al., 2021) without trigger annotations. Text normalization like lemmatization and expanding contractions is applied to data preprocessing. In all experiments, we use mini-batch adam with a learning rate 3e−5 as the optimizer on Nvidia Tesla V100. The ratio of teacher forcing and other hyper-parameters are selected by grid search in (0,1] with a step 0.1. The training takes 30 epochs without early stop. The detailed implementation can be found in Appendix A. The following BERT-based methods are performed for fair comparison: 1) BERT, 2) GDP-Net (Xue et al., 2021), 3) SimpleRE (Xue et al., 2022), 4) D-REX BERT (Albalak et al., 2022), and 5) TUCORE-GCN BERT (Lee and Choi, 2021). Other approaches that take multiple entity pairs for global consideration cannot directly be compared with TREND but reported as reference.

Results of Supervised Joint Learning
The performance of our TREND model jointly trained on the trigger-available DialogRE dataset is presented in Table 1, where it is obvious that our TREND achieves the best performance in the fair setting. Unlike SimpleRE and GDPNet that need to iteratively refine the latent features or latent graphs, relation prediction in the proposed TREND is straight-forward, making training and inference efficient and robust. Furthermore, D-REX (Albalak et al., 2022) also leverages triggers for relation prediction but performs significantly worse than our simple TREND models in the same setting. Our trained binary gate has about 85% accuracy while the trigger prediction has no more than 50% of exact match. Although our model cannot per-  fectly extract the triggers, the predicted spans can still facilitate relation prediction in our proposed TREND. It demonstrates that our TREND model is capable of identifying potential triggers and utilizing such cues for predicting relations. Note that TREND BERT-Large is for reference, indicating that a larger model has the potential of further improving the performance. The upper-bound of our proposed TREND BERT-Base is 75.3 shown in the last row of Table 1, where the ground truth triggers are inputted in the relation predictor. This higher score suggests that our TREND model still has a room for improvement and the proposed model design is well validated.

Results of Transfer Learning
Due to the lack of trigger annotations in DDRel, our TREND model focuses on transferring the trigger-finding capability learned from DialogRE to DDRel. We compare our proposed TREND with two models, which are not designed for transferring across different relation extraction datasets, so they are directly trained on the DDRel data. Table 2 presents the performance achieved on DDRel evaluated in session-level and pair-level settings, where session-level relation extraction is given a partical dialogue the entity pair is involved in and pair-level is based on a full dialogue (Jia et al., 2021). 2 All scores are much lower than ones in DialogRE due to the higher difficulty of this dataset. The obtained improvement compared with the BERT baseline is larger when the longer dialogue contexts as the input; that is, pair-level improvement is more than session-level one. The probable reason is that extracting key evidences for predicting relations is more important to overcome information overload. Furthermore, we report the performance of the current state-of-the-art (SOTA) relation extraction model, TUCORE-GCN, on the DDRel dataset. 3 It can be found that our proposed method can effectively transfer the capability of capturing triggers from DialogRE to DDRel, and outperform TUCORE-GCN in most cases, achieving a new SOTA performance in DDRel.
Suprisingly, TREND BERT-Large does not outperform TREND BERT-Base , implying that TREND BERT-Base already has enough good capability of capturing triggers and can generalize to another dataset (DDRel) and a new relation set.

Ablation Study
Because our trigger finding module contains a binary classifier deciding the existence of explicit triggers and a trigger predictor extracting trigger spans, we examine the effectiveness of the binary gate. By removing the binary gate, the performance is consistently degraded shown in Table 2, further demonstrating the effectiveness of the designed trigger-finding module in our TREND model.

Generalization of Unseen Relations
To further investigate if our trigger-finding capability can generalize to different relations, we categorize all relations into seen and unseen relations based on the relation similarity between the two datasets shown in Table 3, and show the sessionlevel performance in Table 4. It can be seen that our proposed TREND is capable of transferring triggerfinding capability from DialogRE to DDRel, even DDRel does not contain trigger annotations. More importantly, our learned trigger-finding capability is demonstrated general to diverse relations, because TREND achieves better results for not only seen but also unseen relations whose triggers never appear in the DialogRE data. We qualitatively analyze the predicted triggers of unseen relations,   where TREND extracts a dirty word ("fxxk") and a word "client" as triggers for unseen relations "opponent" and "professional contact" in DDRel respectively. The full samples can be found in Table 6. It shows the effectiveness and generalizability of our proposed TREND model towards practical usage.

Qualitative Study
The predicted triggers and relation for DialogRE and DDRel datasets are presented in Table 5 and Table 6 respectively. Note that the triggers are not annotated in DDRel. It can be found that TREND can extract explicit cues as triggers not only for the seen relations, which are similar to relations in DialogRE, but also unseen ones.  Child-Parent father S1: Fuck me! S2: Want a drink? Okay... I'm not good at this sort of thing, but we don't have a lot of time, so I'll just go ahead and get started. Argument Relation (Unseen) Trigger (S1, S2) Opponent fuck S1: I'm Joe Galvin, I'm representing Deborah Ann Kaye, case against St. S2: I told the guy I didn't want to talk to... S1: I'll just take a minute. Deborah Ann Kaye. You know what I'm talking about. S2: No. S1: He's the Assistant Chief of Anesthesiology, Massachusetts Commonwealth. He says your doctors, Towler and Marx, put my girl in the hospital for life. And we can prove that. What we don't know is why. I want someone who was in the O.R. S2: I've got nothing to say to you. S1: You know what happened. S2: Nothing happened. S1: Then why aren't you testifying for their side? I can subpoena you, you know. I can get you up there on the stand. S2: And ask me what? S1: Who put my client in the hospital for life. S2: I didn't do it, Mister. S1: Who are you protecting, then? S2: Who says that I'm protecting anyone? S1: I do. Who is it? The Doctors. What do you owe them? S2: I don't owe them a goddamn thing. S1: Then why don't you testify? S2: You know, you're pushy, fella... S1: You think I'm pushy now, wait 'til I get you on the stand... S2: Well, maybe you better do that, then. Argument Relation (Unseen) Trigger (S1, S2) Professional Contact client showing the great potential of improving explainability without rationale annotations.