{'fraction_non_alphanumeric': 0.02804344302155941, 'fraction_numerical': 0.013292267790565733, 'mean_word_length': 4.776685393258427, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Relation Extraction with Weighted Contrastive Pre-training on Distant Supervision

Contrastive pre-training on distant supervision has shown remarkable effectiveness for improving supervised relation extraction tasks. However, the existing methods ignore the intrinsic noise of distant supervision during the pre-training stage. In this paper, we propose a weighted contrastive learning method by leveraging the supervised data to estimate the reliability of pre-training instances and explicitly reduce the effect of noise. Experimental results on three supervised datasets demonstrate the advantages of our proposed weighted contrastive learning approach, compared to two state-of-the-art non-weighted baselines.


Introduction
Relation extraction (RE) is the task of identifying the relationship between entities mentioned in the text, which can benefit many downstream tasks such as question answering and knowledge base population. Since most of the existing RE models (Zhang et al., 2020;Zeng et al., 2020;Wang and Lu, 2020;Zhong and Chen, 2021) are trained on the labeled data, the amount of training data limits the performance of supervised RE systems. To tackle this problem, recent work leverage a semi-supervised distant supervision (DS) (Mintz et al., 2009;Lin et al., 2016;Vashishth et al., 2018;Chen et al., 2021) approach to generate abundant training data by aligning knowledge bases (KBs) and raw corpora. However, distantly supervised relation extraction (DSRE) inevitably suffers from wrong labeling noise. Introducing a robust framework that utilize both the abundant but noisy data from DS and the scarce but accurate data from human annotations becomes a new research line to improve RE systems.
Recent works (Baldini Soares et al., 2019;Ormándi et al., 2021;Peng et al., 2020) propose a two-stage RE framework that they first design an RE oriented task to pre-train BERT on DS data and  then fine-tune on human annotated (HA) datasets. Peng et al. (2020) use Wikipedia articles as the corpus and Wikidata as the KB in the pre-training stage to construct the DS data, and they introduce a contrastive learning based method to pre-train BERT on the generated DS data. Given an anchor instance with a specific relation in the DS data, their contrastive learning method randomly selects one positive sample holding the same relation and maximizes the similarity between the anchor and positive sample. Meanwhile, their method randomly selects multiple negative samples holding different relations from the anchor and minimizes the similarity between the anchor and negative samples. The results show that their RE oriented pre-training can effectively improve the final performance of the RE task on various target datasets.
However, in their pre-training stage, they ignore the intrinsic wrong labeling noise in the generated DS data. Since their method relies on the DS labeled relation types to sample positive and negative instances, the noisy labeling problem leads to unreliable samples in Figure 1, potentially limiting the pre-training stage's effectiveness. To better utilize DS data, we propose a novel weighted contrastive learning framework to both use the abundant DS data and tackle the inevitable DS noise. First, we train a relation classifier on the HA dataset and leverage the classifier to predict the relation type of instances in the DS data. Then for each DS instance, based on the output of the classifier, we can compute the confidence score to measure the reliability of its labeled relation type. Finally, we introduce weights based on computed confidence scores into the contrastive learning loss to focus more on reliable instances while less on noisy ones.
Besides, distant supervision relies on the existing KBs to align raw corpora. To alleviate the need for KBs, we propose a new strategy to extract a triplet set from the HA dataset for generating DS data. We also include a KB derived DS dataset in our experiments to show that our proposal can still work well for regular DS.
In conclusion, we propose a weighted contrastive pre-training approach for supervised relation extraction and introduce its details in Section 2. Then we perform the experiments on three datasets to compare our proposed method with existing baselines in Section 3.

Overview
We show the overview of our proposal in the Figure 2. We start from generating the DS data relying on the HA dataset. Then in the first stage, we introduce a weighted contrastive learning method by leveraging the HA data to estimate the reliability of DS instances for contrastive pre-training. In the second stage, we further fine-tune our pre-trained model on the HA dataset.

Distantly Supervised Dataset Construction
Since DS uses existing knowledge bases to generate training data, in the case that we have no proper existing KBs in some domains but only the annotated dataset, we first extract all entities based on each sentence, and if any two of them are labeled a relation type, they will generate a triplet with a particular relation. Otherwise, they will still generate a triplet but labeled NA (no relation). After constructing the KB, we can extract sentences containing two entities of each triplet from raw corpora.
To balance the number of sentences extracted by each triplet, we also add an upper bound 100 to the number of extracted sentences.

Two-stage RE Framework
Instance representation In our pre-training stage, we use BERT to obtain the representation for each input instance. For the input format, we follow PURE (Zhong and Chen, 2021) by adding extra special markers to mark the beginning and the end of two entities. For example, given an instance x: "Joe Biden is the president of America.", the input sequence is "[CLS] [H_CLS] Joe Biden [H_SEP] is the president of [T_CLS] America [T_SEP].
[SEP]". Denote the k-th output vector of the BERT encoder as h k . Assuming i and j are the indices of two beginning entity markers [H_CLS] and [T_CLS], we define the instance representation as: where ⊕ stands for concatenation. Then we use the instance representation for the further reliability estimation and the weighted contrastive learning in the pre-training stage.
Reliability estimation With the instance representation, we first fine-tune BERT on the HA dataset as a supervised RE task. Then with the trained relation classifier F, we can make predictions on each instance in the DS data. Given an input instance x with DS labeled relation r, we can derive the confidence score c to estimate its reliability by: where R is the set of all relation classes, and F(x, r) computes the output of our relation classifier on the labeled class r. Through this approach, we can estimate the reliability of the labeled relation for each DS instance by its corresponding confidence score.
Stage 1: DS weighted contrastive pre-training Contrastive learning aims at maximizing the similarity between a given instance and its positive samples while minimizing the similarity between the given instance and its negative samples. As for existing work, Peng et al. (2020) where G is the number of bags in one batch, and the labeled relational triplets are different from each other. Each bag B is constructed by a certain relational triplet (e 1 , r, e 2 ) with all instances x inside satisfying this triplet. Moreover, each instance comes along with a confidence score c estimating its reliability: where N i denotes the size of bag B i . Then the weighted contrastive learning loss of an anchor instance x j in the bag B i is: where cos(·) denotes the cosine similarity between two instance representations, x k denotes the representation of a positive instance sampled from the same bag, and r m = r j denotes that negative samples x m are selected from all instances in the batch that are labeled a different relation from x j . We follow (Khosla et al., 2020) to incorporate multiple positive instances sampled from the same bag. T denotes a scaling temperature.
With the help of confidence scores, the model will focuses on more reliable instances while ignoring unreliable instances which keep paces with our goal to utilize reliable DS data.
Besides, to inherit the ability of language understanding from BERT and avoid catastrophic forgetting, we also adopt the masked language modeling (MLM) objective from BERT.
Eventually, we define our final pre-training loss: Stage 2: Supervised relation extraction We then fine-tune the pre-trained model on HA datasets with state-of-the-art (SOTA) methods. For i2b2 2010VA we follow BLUEBERT (Peng et al., 2019) by treating the relation extraction task as a sentence classification and replacing two named entities in the sentence with predefined tags. For the other two datasets, we follow the encoding method of PURE (Zhong and Chen, 2021) as introduced at the beginning of Section 2.3.

Setup
HA and DS datasets We evaluate our approach on three HA relation extraction datasets: i2b2 2010VA, ACE05, and Wiki20m. Table 2 shows the statistics of each dataset. The i2b2 2010VA is a medical domain RE dataset while other two datasets are collected from general domains. We generate the DS data for i2b2 2010VA and ACE05 from corresponding raw corpora. Meanwhile, Wiki20m is a regular KB based distantly supervised RE dataset containing both DS data and HA data and it is worth noting that we intend to show that our method can also work well on existing DS datasets. Table 3 shows the statistics of DS data.
Baselines We have a naive baseline by directly fine-tuning (FT) on each dataset as a supervised RE task. We set two two-stage framework baselines: the first one is to use the SOTA method RE-Context-or-Names (RECN) (Peng et al., 2020)    in pre-training, and the second one is to use the SOTA DSRE method Contrastive Instance Learning (CIL) (Chen et al., 2021) in pre-training.
Implementation details To further confirm the effectiveness of our proposal, we also conduct the experiments in the low-resource setting by randomly selecting 25% of the full HA data to construct the DS data for pre-training and finally finetune on this 25% HA data. Refer to Appendix A for other implementation details.

Further Analysis
We find that ACE05 contains many pronoun entities, for example, "He lives in America.". As pronoun entities such as "He" naturally come along with much more severe noise in DS, we also conduct extra experiments by removing sentences containing pronoun entities in ACE05 and the corresponding DS data to confirm the effect of pronouns. After removing pronoun entities in ACE05, as shown in the Table 4, our model outperforms all baselines including FT, which indicates that pronoun entities bring mishandled noise in the pretraining stage and limit the effect of our DS data construction approach.

Conclusions
We introduce a weighted contrastive pre-training method by leveraging the HA dataset to estimate the reliability of instances in the abundant DS data. To alleviate the need for KBs, we also propose to construct DS data based on the triplets derived from the HA dataset for pre-training. Experimental results demonstrate that our proposed method outperforms SOTA work on target HA datasets.
One limitation of this work is that our method still needs a certain amount of HA data to achieve sufficient size of DS data and accuracy of reliability estimation. We leave it for future work to reduce reliance on HA data.

A Implementation Details
During the construction of DS data, we use the preprocessing tool NLTK to split raw corpora into sentences.
We use bert-base-uncased (Devlin et al., 2019) as the base encoders for ACE05, ACE05 (no pronouns) and Wiki20m, for a fair comparison with previous works. We also use bluebert (Peng et al., 2019) as the base encoder for i2b2 2010VA, since the SOTA performance is achieved based on this effective medical domain BERT.
For baseline models, we modify their official implementations to fit our experiments and follow the model settings in their papers. For our proposed method, the primary hyperparameters in the experiments are batch size, bag size, and contrastive learning temperature that directly influence the weighted contrastive learning loss, and we show our searching ranges and best values in Table 5.
We used 8 NVIDIA A100 for pre-training and 2 NVIDIA RTX3090 for fine-tuning.