{'fraction_non_alphanumeric': 0.03269318701417467, 'fraction_numerical': 0.0075826855662246605, 'mean_word_length': 4.847816399286987, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Investigation of BERT Model on Biomedical Relation Extraction Based on Revised Fine-tuning Mechanism

With the explosive growth of biomedical literature, designing automatic tools to extract information from the literature has great significance in biomedical research. Recently, transformer-based BERT models adapted to the biomedical domain have produced leading results. However, all the existing BERT models for relation classification only utilize partial knowledge from the last layer. In this paper, we will investigate the method of utilizing the entire layer in the fine-tuning process of BERT model. To the best of our knowledge, we are the first to explore this method. The experimental results illustrate that our method improves the BERT model performance and outperforms the state-of-the-art methods on three benchmark datasets for different relation extraction tasks. In addition, further analysis indicates that the key knowledge about the relations can be learned from the last layer of BERT model.


I. INTRODUCTION
In the past few decades, biomedical literature has been growing explosively and its interdisciplinary nature makes biomedical text mining a unique part of modern biomedical research. Among text mining tasks, relation extraction (RE) plays a crucial role in extracting the structure of the text. The task of relation extraction in biomedical domain is to identify semantic relationships between biomedical entities mentioned in the text. For example, protein-protein interaction is a RE task that involves the extraction of the interacting proteins from the literature. RE allows information available in unstructured form in the literature to be stored in a structured form that can be used in many downstream applications like database curation, question answering system, text summarization etc.
Many different methods have been proposed to solve the RE problem [1]- [4]. Recently, utilizing neural pre-trained language models has been shown to be an effective way to improve model performance of natural language processing (NLP) tasks including RE [5]- [8]. These methods can make best use of a large amount of unlabeled data in an unsupervised pre-training phase to build language representation. This phase can be followed by supervised fine-tuning on taskspecific datasets. In this work, we will employ the BERT [7], a language representation model based on bidirectional Transformer [9], to solve the relation extraction problem. In the pre-training stage, BERT learns general language representation for both the words and the text sequence through two unsupervised tasks. This task-agnostic representation is applied for the specific tasks via supervised learning on taskspecific datasets in the fine-tuning stage.
Since the development of BERT [7], previous works on relation extraction and other classification tasks have all utilized the suggested approach of using a special token called the classification token (CLS) in the topmost layer. Thus, during fine-tuning for classification tasks, all information is funneled through a part of the top layer during back-propagation. In this work, we will explore the potential of utilizing all the information in the last layer to improve model performance on relation extraction tasks.
Our work is partly driven by an observation in [10], where the nature of information represented in various layers was studied. The authors found that BERT models appeared to mimic traditional NLP approaches where low-level morphological and syntactic aspects are represented in lower layers and higher level and more semantic nature of text is learned in higher layers of BERT model. Since relation extraction task is clearly semantic in nature, we hypothesize that important information for this task will reside in top layers of the BERT model. We therefore wish to investigate whether all the information in the top layer of the BERT model could be fruitfully used to improve the performance of RE models.
To test our hypothesis, we employ two different methods to summarize the information in the last layer: Long Short Term Memory (LSTM) model [11] and attention mechanism [12], and then include the summarized knowledge in a new fine-tuning process. We test these methods on three widely studied RE tasks in biomedical domain: the protein-protein interaction (PPI) [13], the drug-drug interaction (DDI) [14], and the chemical-protein interaction (ChemProt) [15]. Our results show that utilizing all the information from the last layer boosts the model performance on all three tasks and furthermore, we now achieve the state-of-the-art results on the three benchmark datasets.
The rest of this manuscript is organized as follows. We discuss some related works and background in Section 2. In Section 3, we describe the details of methodology and experimental settings, followed by the evaluation results and discussion in Section 4. We conclude in the last section.

II. RELATED WORK AND BACKGROUND
We first present some related works in this section. Then we briefly introduce BERT model, its pre-training and fine-tuning process.

A. Related Work
Extracting various of relations in BioNLP domain has attracted considerable attentions in recent years with deep learning methods and high-quality word representation (word embedding) showing advancements in performance [16]- [18]. Both convolutional neural networks (CNN) and recurrent neural networks (RNN) [3], [19], [20] have achieved notable results on relation extraction tasks based on word embedding in this field before the emerging of utilizing language model.
An advantage of language model methods is that they can leverage a large amount of unlabeled data to learn the context-dependent and universal language representation. The pre-trained language model can be applied on downstream tasks through fine-tuning and many language models have been proposed such as ELMo [6], GPT [21] and BERT [7]. Peters et al. proposed a contextualized word representation from a pre-trained language model (ELMo) and significantly improved the performance on six tasks of question answering, textual entailment and sentiment analysis [6]. The Generative Pre-trained Transformer (GPT) [21] was based on a unidirectional language model and has shown its effectiveness on benchmarks for natural language understanding. BERT (Bidirectional Encoder Representations from Transformers) model [7] was designed to learn language representation in bidirectional way and was demonstrated to obtain state-ofthe-art performance on tasks such as question answering and language inference.
Usually, pre-trained BERT model for general domain generalizes poorly in a specific domain like biomedical domain. In order to facilitate the biomedical research, several BERT models have been adapted for the biomedical domain such as BioBERT [22], SciBERT [23], BlueBERT [24]. These models are pre-trained on different types of biomedical corpora and have been shown to achieve state-of-the-art results on various biomedical tasks. BioBERT [22] was pre-trained on PubMed abstracts and PMC full-length articles, BlueBERT [24] used PubMed abstract and MIMIC-III clinical notes [25] as the pretraining data, and a sample of 1.14M papers from Semantic Scholar [26] were used in SciBERT model [23]. In this work, we will first select the best BERT model on our relation extraction tasks and then employ that model in the following experiments.
However, most of previous works only use the output of classification token ([CLS]) in BERT model for classification problem and ignore other outputs, which contains useful information. In the work [27], the authors explored the methods of extracting knowledge from the intermediate layers in BERT and demonstrated the advancement on aspect based sentiment analysis and natural language inference tasks. Since residual connections are employed in the Transformer, all important information in intermediate layers should be transmitted to the last layer of BERT model during training. We will compare this method with our approach of utilizing all the information from the last layer in BERT model.
In addition, the work [10] found that the BERT model represents the steps of the traditional NLP pipeline in a sequential way, which means the BERT model captures the linguistic information in the sequence of part-of-speech (POS) tagging, parsing, NER (Named Entity Recognition), semantic roles, then coreference. Their experiments also showed that the representation for relation classification continues to improve up to the highest layer of the BERT model, indicating that the last layer contains important information for the relation extraction tasks.
Our method evaluation will be conducted on three relation extraction tasks: PPI [13], DDI [14], and ChemProt [15]. For PPI, it is probably the most widely studied relation extraction task in biomedical domain because its key role in understanding biological processes. Extracting relation of DDI will help prevent adverse effects from drug combinations. ChemProt relation extraction will benefit precision medicine by finding the affected proteins by a specific chemical. Dozens of systems have been designed for these tasks like rule/kernelbased methods [28]- [30], neural network methods [3], [31], [32], language representation model methods [22], [24]. As far as we know, the state-of-the-art system on AIMed corpus for PPI task is based on deep residual convolutional neural network [3] while the best results on DDI and ChemProt tasks are obtained using the BERT model [22], [24].

B. Background
BERT [7] is a bidirectional Transformer model [9] for language representation. Using a "masked language model", BERT is able to learn bidirectional representation (both left and right context) of a word. The application of BERT usually includes two steps: utilizing pre-training to gain general knowledge and employing fine-tuning to acquire task-specific knowledge of a task.
Pre-training of BERT: BERT is pre-trained using two unsupervised tasks: masked language model (MLM) and next sentence prediction (NSP). The pre-training procedure usually involves a large amount of unlabeled data. Originally, BERT was designed for general purpose and it was pretrained on English Wikipedia and BooksCorpus. However, each domain has its specific knowledge and sometime it is impossible for the general BERT to gain such knowledge. For example, in biomedical domain, the biomedical entity names can not be represented well based on the general BERT model since its pre-training corpora are short of such names. In this paper, we will consider three BERT models that are adapted to the biomedical domain (BioBERT [22], SciBERT [23] and BlueBERT [24]) and select the best one for the experiments on the new fine-tuning process.
Fine-tuning of BERT: The pre-trained BERT model can be fine-tuned for various downstream tasks. Given a task, we just need to append an additional output layer on top of the BERT output and then feed the task-specific inputs and outputs into BERT model and fine-tune the BERT parameters like regular model training.

III. METHODOLOGY
In this section, we first define the relation extraction problem. Then, we propose the method of adding a new module to summarize the output information in the last layer during finetuning. The evaluation datasets and metrics, the fine-tuning parameters and pre-processing of data for BERT model are also given in this section.

A. Relation extraction
Relation extraction task can be seen as a classification problem. For example, for the DDI relation extraction task, a sentence instance with two marked drug mentions is considered one of the four relations (ADVICE, EFFECT, INT, MECHANISM) if the sentence expresses drug interaction between the two drug occurrences, otherwise it will be seen as negative. Formally, for a relation R and its label set L, our model needs to predict the probability of each label P (L|T ok 1 , T ok 2 , . . . , T ok n , Et 1 , Et 2 ) based on the sentence text S = T ok 1 T ok 2 . . . T ok n and two entities Et 1 , Et 2 in the sentence. BERT model can be applied on this relation extraction classification problem.
B. New fine-tuning process of BERT model As mentioned above, a BERT model for classification problem is usually based on the [CLS] token of the last layer and other outputs of the last layer are abandoned. So this method might discard some useful information from the last layer. The [CLS] token is used to predict the next sentence (NSP task) during the pre-training, which usually involves two or more sentences, but the input of our relation extraction tasks only contain one sentence. This indicates that the [CLS] output might ignore important information about the entities and their interaction because it is not trained to capture this kind of information. The fine-tuning step on task-specific datasets compensates this information loss on relations to a certain extent, but utilizing all information might yield a better representation than the use of partial information (only the [CLS] output). As it is also shown in [10], higher layer outputs yield better representations for the relation classification task. In this paper, we will design a mechanism to utilize all the information in the last layer during the fine-tuning of BERT model.
To include the information from other outputs of the last layer, we propose to add an new module to summarize these outputs and then concatenate the summarized information with the [CLS] output as the extra step for fine-tuning. Specifically, we explore two methods of summarizing the information in the last layer: Long Short Term Memory (LSTM) model and attention mechanism. The model architectures of these two methods are illustrated in Fig 2. Formally, given the hidden states (with dimension of H) of the last layer (L-th layer) CLS is used for classification in the original BERT model. Here we will try to add the summarized information from the sequence of other outputs, a.k.a., h L = {h L 1 , h L 2 , ..., h L N } (We ignore h L SEP since it is a separation token for different sentences) and two different approaches are employed: LSTM sequence model and attention mechanism.
1) A natural idea to model the sequence of the last layer is to use recurrent neural network and LSTM model is our first choice since its advantages on long sequence. The output of last LSTM cell will be seen as the representation of our sequence. Formally, .., N 2) Another choice of summarizing sequence data is the attention mechanism. Specifically, we can utilize additive attention to combine the sequence by assigning a weight for each output.
where K H×1 are learnable parameters.   Finally, we feed this representation to a softmax layer to predict the label: f are also learnable parameters and C is the number of classes (categories) in the classification task.
Meanwhile, the contribution of [CLS] token for the classification task needs to be explored. So we experiment with another fine-tuning process without [CLS] token. Formally, we only take h = [O] as the final representation of the input and utilize the same softmax layer p = sof tmax(W f h + b f ) for prediction. The performance of these two new fine-tuning processes can shed light on the roles of the two parts (the [CLS] token and sentence outputs) of the last layer for classification problem.

C. Datasets and evaluation metrics
To measure the effectiveness of the new fine-tuning mechanism, we will evaluate it on three benchmark datasets. Table I shows the statistics of these datasets. For PPI task, we will utilize the most well-known PPI relation corpus: AIMed [33]. It is obtained by annotating 750 abstracts from Medline and it contains 1,000 positive (interacting protein pairs) and 4,834 negative (non-interacting protein pairs) instances. Since there is no standard training and test set for this dataset, we will employ 10-fold cross-validation on it.
The corpus for ChemProt task contains a training set of 1,020 abstracts, a development set of 612 abstracts and a test set of 800 abstracts from PubMed [15]. We will use the standard training and test set to evaluate the model performance.
For the DDI corpus, it has 792 documents from DrugBank database and 233 abstracts from Medline [14]. Similar to ChemProt, we will use the standard split of this dataset for evaluation (624 documents for training, 210 for development and 191 for test).
As for the evaluation metrics, PPI is a binary classification problem, so we will use precision, recall and F1score to measure the performance of the models. However, the ChemProt and DDI tasks are multi-class classification problem. The ChemProt corpus is labeled with six classes: CPR:3 (upregulator, activator, indirect upregulator), CPR:4 (downregulator, inhibitor, indirect downregulator), CPR:5 (agonist, agonist-activator, agonist-inhibitor), CPR:6 (antagonist), CPR:9 (substrate, product of, substrate product of) and negative. In DDI corpus, there are five labels for the instances: ADVICE, EFFECT, INT, MECHANISM and negative. For ChemProt and DDI tasks, we evaluate the model utilizing micro (i.e., with weighted mean) precision, recall and F1 score on the non-negative classes as other people report their results in the same way.

D. Fine-tuning hyper-parameters
During the fine-tuning of the models, we use learning rate of 2e-5, batch size of 32, training epoch of 10 and max sequence length of 128.

E. Data pre-processing
In relation extraction tasks, the input for the BERT model contains two parts: the text and the entities in it. In order for   model of applying attention mechanism on the outputs of last layer. To the best of our knowledge, the state-of-the-art systems on the three tasks are: [3] for PPI; [22] for ChemProt; [24] for DDI.
BERT model to recognize the entities in the sentence, we follow the standard pre-processing way for relation extraction by replacing these entity names with predefined tags. Specifically, we replace all the protein names with @PROTEIN$, drug names with @DRUG$, and chemical names with @CHEM-ICAL$. In Table II, we show some examples from the three tasks, in which the entities are replaced with the special tags. In order to alleviate the out-of-vocabulary problem, BERT uses WordPiece embedding [34] and each input word will be split into subwords from WordPiece vocabulary. In addition, following the original BERT input representation, we add the segment and position information of the tokens in the input sentence through the segment embedding and position embedding.

IV. RESULTS AND DISCUSSION
In this section, we first discuss the model selection from the three BERT models for biomedical domain. Next, the results of new fine-tuning process are provided. At last, more discussions on attention weights are given to help us better understand the information summarization in attention mechanism.
As mentioned earlier, there are three adapted BERT models for biomedical domain: BioBERT, SciBERT and BlueBERT. The difference between the three models adapted to the biomedical domain is the text used for the pre-training. After fine-tuning on the corpora of the three tasks, BioBERT generally obtains good results (Please refer to Table VI in the Appendix for the details of the performance of the three models). Therefore, we will report our experimental results of new fine-tuning process based on BioBERT.

A. Model performance after utilizing all the information from the last layer
To provides context for the performance of the proposed method, we first report the state-of-the-art (SOTA) results and the performance of original BioBERT model on the three tasks in the first two rows of Table III. As shown in the table, the BioBERT model outperforms the state-of-the-art method on PPI task, but it is not up to par with the current SOTA methods on DDI and ChemProt tasks.
In the last two rows of Table III, the model performance of including the knowledge of the last layer is illustrated. Clearly, the use of information in the last layer leads to improvement for all three tasks. The model of utilizing attention mechanism on the last layer achieves the best performance on all the tasks, with 1.5%, 2.3% and 1.1% F1 score improvement on PPI, DDI and ChemProt respectively. We also see that the use of attention mechanism outperforms the LSTM method on summarizing the information of last layer.
Furthermore, we achieve the state-of-the-art performance on all three tasks after adding the attention mechanism on the last layer. Specifically, our method leads to 4.9%, 0.8% and 0.3% F1 score improvement on PPI, DDI and ChemProt respectively.

B. Other fine-tuning mechanisms
The results of new fine-tuning of BERT model that includes output information from both [CLS] token and the other tokens of the last layer have shown that this method achieves better performance than original fine-tuning mechanism. In this work, we also experiment with the method of fine-tuning without [CLS] output to explore the contribution of [CLS] token. In Table IV, the two rows in the middle (BioBERT+LSTM LL* and BioBERT+Att LL*) illustrate the model performance of utilizing the outputs of last layer without [CLS] token. The first two rows are only repetitions of the results from Table  III

C. Visualization of attention weights
The information acquired by attention mechanism from the last layer has been shown to improve the model performance.
In this section, we will visualize the attention weight distribution on the words of sentences and examine the words the model is paying more attention to.
We select an example from each of the three tasks that were misclassified by the original BioBERT model but are correctly predicted by the new method. In Fig 3, we visualize the weights based on the darkness of the color, with words in darker color having larger weights.
We can see that the attention mechanism is giving large weights on the important words that indicate the relation between the entities. Typically, the word that is used to express the relation in a sentence is called "trigger word" of the relation. The attention mechanism is gaining the knowledge of "trigger word" in our tasks. For example, in the PPI sentence (Fig 3a), the words of entity token (@PROTEIN$) and the word "activates" (trigger word of PPI) have larger weights than other words. Similarly, the trigger words "inhibit" and "mediate" have larger attention weights in the DDI and ChemProt examples respectively. Recall these sentences were not correctly predicted when attention was not used to highlight the information from the "trigger word", which also demonstrates the importance of the information from the last layer.

D. More analysis of attention weights
After providing some examples of attention weight distribution in the sentences, it is useful to conduct an analysis on the attention weights and investigate the overall weight distribution on the words in the corpora.
Previously, we have shown that the attention mechanism will learn relatively larger weights on trigger word of the relations using three examples. In order to understand the attention weight especially on trigger words in a more general way, we need to consider all the words in the corpora. Given the fact that the trigger words usually appear near the entities of the relation, we collect all the weights of three words around the entities in the positive instances and take the average of the weights to acquire the global attention for each word. Considering the different forms of the words, we utilize Porter's stemmer [35] to remove the morphological affixes from each word and only use the word stem for the global attention weight calculation. For instance, the stem for the word "activate" is "activ", and many other words like "activates" and "activation" have the same word stem.
In Table V, we show the top 10 words that have large learned weight from the attention mechanism for the dataset of each task. Among the top words, we can see that most of them are "trigger words" of the relations, which indicates that   4: Comparison of trigger word weights in positive and negative instances. The first two words (associate and interact) are from PPI task, the middle two words (decrease and inhibit) are from DDI task and the last two words (stimulate and regulate) are from ChemProt task.
the attention mechanism is learning the knowledge of relations from the last layer of BERT model.

E. Trigger word weight comparison between positive and negative instances
We have seen that the attention mechanism is giving relatively large weights on trigger words of relations on positive instances. For the negative instances, small weights should be learned on trigger words since there is no relationship between the entities. Therefore, we compare the trigger word weights between positive and negative instances and give six examples in Fig 4 from the three tasks. We can see that the learned trigger word weights in positive instances are much larger than the weights in the negative instances. This demonstrates that the attention mechanism can automatically learn the difference between positive and negative instances.

V. CONCLUSION
In this paper, we propose a new fine-tune process for BERT model to utilize all the outputs from the last layer. We verify the effectiveness of the method on three relation extraction tasks: PPI, DDI and ChemProt. The experimental results illustrate that the proposed method achieves state-ofthe-art performance on all three tasks. Also, the visualization and analysis of the weights in attention mechanism also demonstrates that our method is learning the key knowledge of the relations in the sentence to help the relation extraction.
In the future, We will continue our investigation on extraction of other relations between biomedical entities. We will also investigate how our method can help other tasks in biomedical domain.

APPENDIX COMPARISON OF BERT MODELS IN BIOMEDICAL DOMAIN
We provide the performance of BioBERT, SciBERT and BlueBERT in biomedical domain on the PPI, DDI and ChemProt tasks. The best result on each metric is highlighted with bond font in Table VI