{'fraction_non_alphanumeric': 0.02955324765029479, 'fraction_numerical': 0.008116042134346399, 'mean_word_length': 5.039630512514899, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A New Entity Extraction Method Based on Machine Reading Comprehension

Entity extraction is a key technology for obtaining information from massive texts in natural language processing. The further interaction between them does not meet the standards of human reading comprehension, thus limiting the understanding of the model, and also the omission or misjudgment of the answer (ie the target entity) due to the reasoning question. An effective MRC-based entity extraction model-MRC-I2DP, which uses the proposed gated attention-attracting mechanism to adjust the restoration of each part of the text pair, creating problems and thinking for multi-level interactive attention calculations to increase the target entity It also uses the proposed 2D probability coding module, TALU function and mask mechanism to strengthen the detection of all possible targets of the target, thereby improving the probability and accuracy of prediction. Experiments have proved that MRC-I2DP represents an overall state-of-the-art model in 7 from the scientific and public domains, achieving a performance improvement of up to compared to the model model in F1.

In addition, by locating the answers directly instead of tagging them verbatim, the probability of long entities with more information disrupted inside is relieved, which improves the quality of extraction.
However, there are still two significant problems in the relevant entity extraction models based on QA or MRC.
On the one hand, there is a lack of focus in the text during the reading stage (not all parts are related to the target entity) and a lack of in-depth interaction between question and context, which is not in line with the standards of human reading comprehension. We usually pay attention to the paragraph whose content is closest to the given question, fully connect (i.e., interact) the contextual information to better understand the context, and then locate the possible answers. However, the existing related models only concatenate the paragraph and question to perform shallow unified processing [13]- [16] or merely increase the number of questions [17], so that the model can only give the direct answer after a simple extensive reading, which limits its comprehension ability. Therefore, when extracting Task, Method, and the other type of phrasal entities with a higher understanding requirement in scientific literature, as shown in Fig. 1(a), the performance is poor.
On the other hand, the problem of incomplete or inaccurate positioning of the answers (i.e., target entities) at the extraction stage, resulting in omission or misjudgment. These missing entities sometimes are essential, and misjudged ones bring us wrong information. Therefore, fully and accurately positioning all the answers is as important as the reading process. However, relevant models get the answer positions by using the output 1D prob-distribution [13], [14], [17] or generating and matching two independent start and end position distributions of the answer [15], [16]. The former cannot fully extract nested entities with intensive information, as shown in Fig. 1(b), and the latter is prone to entity misjudgment.
Aiming at the above problems of related works, an MRC-based entity extraction model, MRC-I2DP is constructed in this paper, which is more effective and more in line with human reading comprehension. By exploiting the proposed gated interactive attention mechanism and 2D probability encoding, the model enhances the entity understanding and positioning respectively, thus improving the overall extraction performance. Concretely, the main contributions of this work are as follows:  A gated interactive attention mechanism is proposed, guided by the norms of human reading comprehension. Firstly, an adaptive gating matrix used in the proposal adjusts the importance of each text part to the answers to provide the focus for the model. Secondly, the multi-level interactive attention calculation is applied to the context and the question to make the model deeply think with the target entity type, enhancing the overall attention on aimed entities and alleviating their semantic interweaving with other words. As a result, the comprehension and recognition ability of the model improved.  A 2D prob-encoding module for answer positioning is proposed, equipped with a new probability function named TaLU and a 2D distribution mask mechanism. The module constructs the 2D position coordinate matrix of the answer, strengthening the boundary detection of all possible entities (especially nested entities) and increasing the extraction recall. Besides, each element in the square matrix uniquely determines an accurate start and end position pair of the potential entity, which avoids mismatching in two independent 1D distributions and improves the extraction accuracy.  An MRC-I2DP model is proposed based on the above gated interactive attention mechanism and 2D probencoding to effectively extract entities from massive texts in the scientific and public domain. The model uses parallel computing to avoid a dramatic increase in time complexity and takes a masking mechanism to filter much noise to prevent the accuracy decline caused by the imbalance between positive and negative samples. Experiments show that MRC-I2DP achieved F1 scores of 80.2%, 91.4%, 85.0%, 70.8%, 62.6%, 88.2% and 88.5% on 5 scientific domain literature datasets: GENIA [18], ADE [19], SOFC [4], SciERC [20], ScienceIE [21] and 2 public domain datasets: ACE2004 [22], ACE2005 [23], which outperforms most state-of-the-art models, achieving a performance boost of 2.1% to 10.4% over baseline models, proved to be of strong entity extraction ability and generalization.

A. Mainstream Methods for Entity Extraction
Entity extraction, first proposed at the Sixth Message Understanding Conference (MUC-6), is a critical technique in natural language processing for automatic information extraction and the basis of other tasks like reasoning and prediction. With the development of deep learning, the model architecture for entity extraction has shifted from the initial rule-and-featurebased construction to the neural network framework. The scope of application is also expanding, covering public domain texts and documents in various scientific fields.
Collobert et al. [1] proposed the first entity extraction model based on neural network, which combines Convolution Neural Network (CNN) with Conditional Random Field (CRF) to extract a string of words tagged as the same entity type by sequence labeling. This idea of word-by-word tagging is still the most popular entity extraction method in current applications. For example, Strakov'a et al. [2] combined the Bidirectional Long Short-Term Memory (BiLSTM) network, CRF, and pre-trained BERT model to extract Organization, Geopolitics, and other types of entities from ACE2004 and ACE2005 datasets. Jiang et al. [3] also used the architecture of BERT+BiLSTM to extract Protein, DNA, RNA, and others from GENIA. In addition, Friedrich et al. [4] capture Materials, Devices, and other types of entities from the SOFC dataset by a combined model of the pre-trained SciBERT, BiLSTM, and CRF. And Sahrawat et al. [5] used similar model structures to extract Task, Processing and other phrasal entities from the ScienceIE dataset.
In recent years, some entity extraction models have been improved based on sequential tagging to better deal with the more complex and diversified knowledge and information. For example, Shibuya and Hovy [6] proposed a Path-BERT model, which regards the tagged sequence as a second-best path, extracting entities iteratively in an outside-to-inside way. Fu et al. [7] proposed a new tagging algorithm named TreeCRF, which extracts entities by modeling a tree structure of their tags. And Kong et al. [8] further incorporated the graph-structured information during tagging.
Another kind of mainstream entity extraction method is span-based classification. All the spans in the text are enumerated and classified to determine whether it is the target entity. For example, Eberts and Ulges [9] proposed a BERTbased span classification model named SpERT to extract entities from ADE and SciERC. Wang et al. [10] further combine CNN on this framework to obtain span representation with local semantic information. Zhong and Chen [11] proposed a span classification model based on SciBERT, which pre-filtered spans with lengths larger than the setting threshold, and Shen et al. [12] added a target detection algorithm called SoftNMS based on span classification to enhance the detection of spans that contain target entities.
Although the above mainstream related models achieved some success in the scientific and public domain, the extraction by tagging word-by-word or enumerating span does not accord with the purposeful reading and information search, leading to a lack of the pertinence to target entity type and the understanding of text content. They also have disadvantages of high time complexity and/or introduce much noise during training, which results in low efficiency. Therefore, under the guidance of human reading and information search norms, this article uses the form of reading comprehension for entity extraction, transforms the tagging and the enumeration into the arxiv 3 way of questioning model to improve the comprehension and extraction efficiency.

B. QA/MRC-based Entity Extraction
Compared with the widely used sequence tagging or span classification methods, the entity extraction model based on QA or MRC acts more like human beings to read and search for the required information and has more advantages. Concretely, the question provides the model with a priori message of the target entity category, which is beneficial for extracting corresponding entities pertinently. Secondly, entity extraction based on QA or MRC helps the model obtain more accurate results after understanding the text content. In addition, directly locating the answer rather than tagging word-by-word improves the efficiency and reduces the probability that long entities with much information break from the inside during extraction.
In the past two years, some researchers have applied the QA or MRC form to entity extraction by transforming target entity types into questions and regarding the entity as the answer to the question, to a certain extent, enhancing the model's comprehension and improving performance. For example, the team from Shannon Technologies [13] proposed a BERT-based Multi-turn QA model to transform the extraction of different types of entities into a multi-round Q&A process and locate the target entity through a BMEO (Begin, Middle, End, and Outside) fully connected layer. Jain et al. [14] also proposed a QA-based framework for entity extraction, which obtains the location distribution of entities through a layer of 1D convolution. Li et al. [15] combined the BERT with the MRC task form, previously designed questions corresponding to each entity type, and got the starting and ending positions of answers through a layer of linear transformation after splicing the text and question representation at the output. Zheng et al. [16] and Zhao et al. [17] used the same model architecture for entity extraction, increased the number of fully connected layers and questions, respectively.
Although the above entity extraction models adopt the task form of QA or MRC, they still lack the focus in the text and ignore the further interaction between questions and contextual information, which is inconsistent with human reading habits. The shallow unified processing or a just numerical increase of questions limits the model's ability to understand. Besides, their way of answer positioning is prone to the omission or misjudgment of target entities. In this paper, MRC-I2DP uses the proposed gated interactive attention mechanism to provide the focus for the model while calculating the multi-level interactive attention between contextual information and question to enhance the comprehension and recognition ability, as well as constructing a 2D coordinate probability matrix to improve the recall and precision of extraction.

III. MRC-I2DP MODEL
The MRC-I2DP proposed in this paper is a new and efficient model for automatic entity extraction, suitable for scientific literature and public domain texts. The training and inference process is in the task form of MRC so that the model requires relative questions or query statements designed for all target entity types in advance. For example, when extracting Neural Network type entities from AI domain literature, the query can be "Please find the neural networks such as CNN, LSTM, GRU or Transformer, BERT, etc." The input of the model is a concatenation of text and a query sentence. Since the role of the query is to provide prepared information of the target entity type, we suggest only keeping the corresponding keywords rather than paying attention to the whole semantic coherence. Thus the above query can be further simplified to "neural network CNN LSTM GRU Transformer BERT" to reduce the length of the input sequence and the amount of computation.
The output of the model consists of three parts: two vector distributions and a probability distribution matrix. The former two indicate the start and end positions of the target entities, while the last is the crucial 2D entity coordinates. Each element in the vector distribution means "the probability of the index predicted as the initial start or end position." and each one in the 2D probability distribution matrix refers to "the probability of the horizontal and vertical indexes of the coordinates being predicted as the start and end position, respectively." The three probability distributions above and the corresponding mask mechanism with a probability threshold T helps the model to obtain the final start and end positions to extract entities from the text.
The MRC-I2DP model takes the proposed 2D probability encoder as the core module, combined with a gated interactive attention mechanism that conforms to the human reading comprehension norms to enhance the understanding and attention of the target entities and the detection of all possible boundaries. In addition, Byte Pair Encoding (BPE) for tokenization and DeBERTa [24] for pre-trained word vectors are used to improve the entity extraction performance of the model. The overall architecture of our MRC-I2DP, as shown in Fig. 2, consists of three main modules.
 The Pre-trained module: conducts a feature extraction of the input after BPE tokenization by deep self-attention to obtain the representation of word vectors with the global semantic information.  Pointer module: implements the preliminary interactive attention calculation on the output representation of the pre-trained module above. The result is mapped to a vector distribution with the same length as the input and then normalized to probability.  2DP-Encoder module: proceeds a further gated interactive attention calculation on context and question from output representation of pre-trained module. The result is mapped to a square matrix of 2D probability distribution with the same length and width as the input and then normalized to probability.

A. The Pre-trained Module
The pre-trained module aims to get the word representations (i.e., vectors) that are injected with the global semantic information, to enhance the model's ability to understand the content of the text. First of all, the input sentence of the text and designed query are concatenated to an input format accepted by the pre-trained module, as shown in Eq. (1).
where the "[CLS]" token represents the semantic information of the whole sentence, {q0, q1, q2, …} and {t0, t1, t2, …} are the query of target entity type and the text sentence, respectively; and the "[SEP]" token is used as the separator. Secondly, to reduce the number of rare words in the text and alleviate the Out-Of-Vocabulary (OOV) problem, the model takes BPE encoding (essentially a subword algorithm based on the frequency statistics of the continuous byte pair) for tokenization. The rare words are decomposed into common affixes and fragments to improve the model's cognitive ability to these obscure domain conceptual term entities.
The input of the pre-trained module is the sentence that went through word tokenization. There are two main reasons why to choose DeBERTa as the pre-trained module. For one, the self-attention score of an entity depends not only on its content but also on the relative position between words. For example, the dependence of the tokens "deep" and "learning" when they appear adjacent is much stronger than that when they are far away, which is particularly prominent in scientific literature. DeBERTa adopts a modified self-attention mechanism based on the separation of content and position representation, which improves the effectiveness of the self-attention scores of entities. For another, an Enhanced Mask Decoder (EMD) in the output layer of DeBERTa alleviates the mismatch between pre-trained and fine-tuning to some extent, improving the robustness of the model for entity extraction in different domain texts.
In this paper, we selected the representation of the last hidden layer in DeBERTa as the output ho of this module: (2) where l is the sequence length of ho after BPE tokenization and d is the dimension of word representation.

B. Pointer Module
The Pointer module is responsible for mapping the word representation ho into a vector of probability distribution with the length of l to obtain the preliminary prediction result of the start or end positions of the target entities. Different from the simple linear mapping in the related work of Li et al. [15] and Zheng et al. [16], Pointer exploit the vector representation of "[CLS]" which contains global information to calculate the further interactive attention between text, query, and contextual information to enhance the concern to each whole target entity. As a result, the extraction performance of some phrasal entities with higher contextual comprehension requirements in the scientific literature is improving. The algorithm of the Pointer module shows in Fig. 3.
The input representation ho first goes through a layer of the linear transformation, in which the activation function is an approximate of Gaussian Error Linear Unit (GELU) with the expression of Eq. (3), to obtain a new representation h as shown in Eq. (4).
Then h interacts with the global semantic vector hcls to get the vector of candidate probability distribution ph. () TaLU is a proposed probabilistic (i.e., normalized) function whose expression shows in Eq. (6). It maps the calculated interactive attention score to the range of (0,1). Since the derivative range of the TaLU function is 4.0 times that of the widely used sigmoid function, it can effectively improve the differentiation of the probability at different positions, which is helpful to the boundary detection of the model.
x e e e     (6) In addition, the input ho also goes through a layer of linear compression (d → 1) to obtain another vector of the candidate probability distribution pc. () The initial probability distribution s (or e) of the start (or end) position as the final output is the mean of ph and pc.

C. 2DP-Encoder Module
The 2DP-Encoder module is devoted to further improves the model's extraction performance for some nested entities with intensive information, as well as some phrasal entities that require contextual understanding in the scientific literature.
First of all, there are two cases of entity nesting. One is the nesting in different entity types. For example, "SiC power device" is a Device type entity in which the nested "SiC" is a Material type entity. Another is the nesting in the same entity types, such as that one start position corresponds to multiple end positions, or one end position corresponds to several start positions. For example, "convolution neural network" and its nested "convolution" are both Algorithm type entities with the same start; "HMG I/Y" and its nested "I/Y" are both DNA type entities with the same end. These two kinds of overlap are common in various domain texts, but existing related models are unable or hard to deal with them. However, 2DP-Encoder furthest enhances boundary detection for all possible nested entities in different or same type by mapping the input ho into a 2D probability distribution matrix with length and width of both l, to improve the recall of nested entity extraction. Secondly, the module also combines a gated interactive attention mechanism that accords with the specification of human reading comprehension proposed in this paper. By adaptively adjusting the importance of each part of the text to the answers (i.e., target entities) through the gating matrix, and fully interacting with the global semantic information, the entity type information in the query, and the contextual information of the text, the model's ability to understand the specified entity type and the overall attention to the target entities further enhance. The algorithm of the 2DP-Encoder module shows in Fig. 4.
On the one hand, the input ho goes through two independent linear transformations, respectively. Two representations: hg and h2D, used to calculate the gating matrix and 2D probability distribution matrix, are obtained.
() (9) Then the hg is mapped to the range of (0,1) after normalization, and the gated weight matrix g is obtained.
The gating matrix g is employed to automatically adjust the importance of different text parts related to target entities for the ho and h2D. The result is a vector representation named hf as Eq. (11) shows, in which the "⊙" represents element-wise multiplication.
(1 ) The following step is the calculation of interactive attention  (12) On the other hand, the vector representation hcls with global semantic also interacts with hg and h2D. The resulting vector distribution mh is expanded and superimposed by rows and columns to generate the second candidate 2D probability distribution matrix Mh. [ To further enrich the information in the 2D probability distribution and achieve more accurate positions of target entities, 2DP-Encoder also considers the distribution information  (14) and (15), respectively. () The final output 2D probability distribution m is the mean of On the one hand, since the model takes BPE word tokenization, some complete obscure words in the input sentence are split into multiple segments. The elements of probability distribution with the index corresponding to the middle segments should be zero as they can be neither the starting positions nor the ending positions of the candidate entities. Besides, since the index of the starting positions cannot be larger than that of the ending ones, the lower triangular elements of the generated 2D probability distribution matrix m need to be zeroed.
On the other hand, to avoid the performance degradation caused by the imbalance between positive and negative samples, the probability threshold of training is set to Tt. We selected the elements greater than Tt from s and e. Their corresponding indexes are as the preliminary starting and ending positions of all possible entities, respectively. (17) Then only a few elements in m whose row indexes belong to ps and column index belong to pe are screened out as the final masked result ml for the coordinates of the extracted entities.
The loss function for training consists of three weighted parts: the loss of s and real start position distribution ys, the loss of e and real ending position distribution ye, as well as the loss of masked 2D probability matrix ml and real position coordinate matrix ym.
s y e y m y (19) The function f(x, y) represents the Binary Cross Entropy (BCE) between predictions and true labels.  (20) λ in Eq. (19) is an adjustable parameter, which represents the proportion of 2D probability distribution in the loss.

E. Evaluating of The Model
In the evaluation stage, we added the length restriction of the target entities on the mask used in training. Thus the elements in 2D probability distribution whose vertical coordinate (i.e., the end position) exceed the horizontal coordinate (i.e., the start position) by more than a maximum entity length of lm are set to zero. Making sure some entities with extra length are filtered to enhance the robustness of the model for answer positioning. The final result p of the 2D coordinates shows below: (21) By using all the start and end positions in p, the corresponding text content can be extracted from the input sentence and decoded (by BPE) as the final target entities.

A. Datasets and Evaluation Metric
In this paper, we choose five scientific literature datasets and two public domain datasets which are most representative and widely used in entity extraction research, to verify the entity extraction ability and generalization of the model, as shown in Table Ⅰ  , and Geographical Entities (GPE). We make an exploration of the above datasets, including the proportion of nested entities, whether it contains phrasal entities with higher comprehension requirements, as well as the shortest, the longest, and the average length of the entities, as shown in Table Ⅱ, for the analysis of experimental results.
The evaluation metric adopts the commonly used Precision (P), Recall (R), and F1 score in related works. The calculation expressions show below.
the number of correct entities extracted P the total number of entities extracted  (22) the number of correct entities extracted R the actual total number of entities  (23) 2 PR F1 PR    (24) To be consistent with other related works, we calculate the F1 score on the ADE and the SOFC datasets by inter-class averaging (Macro) while calculating the results on others by total averaging (Micro).

B. Experimental Equipment and Parameters
We carried out all the experiments in this work on a single 11G GTX 1080 Ti. The sentence length of all datasets is limited to 64 words after tokenization. The training takes the AdamW  Table Ⅲ.
In addition, except that the experiments on ADE datasets use pre-trained word vectors of PubMedBERT [25] specific for biomedical fields, DeBERTa is used for other datasets.

C. Comparative Experiments and Discussion
First, comparative experiments are carried out on seven scientific and public domain datasets for entity extraction to verify the overall performance and generalization of the proposed MRC-I2DP model. The results show in  listed comparisons are the representative works or the state-ofthe-art (SOTA) related researches.
To make the comparison results more persuasive, we considered the models based on two kinds of mainstream entity extraction methods and the models based on the QA/MRC method. This work selected the most representative three described in the Related work section as the baseline models marked with "*" in Table Ⅳ and Table Ⅴ. They are:  BERT + BiLSTM: the most widely used mainstream model based on sequence tagging for entity extraction.  SpERT: a mainstream entity extraction model based on span enumeration and classification, commonly used as the baseline model in related works, proposed by Eberts and Ulges [9].  Multi-turn QA: the first and most representative entity extraction model in the kind of works based on QA/MRC. It is widely used as a comparison in related researches and proposed by Shannon Technologies [13]. The experimental results in Table Ⅳ indicate that the entity extraction performance of the proposed MRC-I2DP model surpasses most of the representative and related SOTA models in the fields of biomedicine, material, AI, and computer science. Showing strong practicability and achieves a significant improvement in F1 compared with baselines by 2.1% ~ 10.4%. As a whole, it fully proves that the model can be effectively applied to literature in the different scientific domains to realize automatic entity extraction.
The experimental results in Table Ⅴ show that, for public domain news texts in which there are also a large number (22.0% ~ 24.0%) of nested entities in non-scientific fields, MRC-I2DP is also better than most representative and related SOTA models. It achieved an F1 increase of 3.7% ~ 4.6% relative to the baselines, proving that our model can also be effectively applied to public domain texts and has a strong generalization.
To help analyze the comprehension ability of MRC-I2DP to the target entities, a comparative histogram of the performance improvement (relative to baselines) in different kinds of scientific literature is constructed in this section, as shown in Fig. 6. It exhibits that the entity extraction performance of computer science and AI domain literature is generally lower than that of texts in other scientific fields. However, our MRC-I2DP model happens to have a significant F1 boost in these fields (up to 7.0% ~ 10.4%). Given this result, we delivered the following explanation. Most of the knowledge and information in scientific fields like biomedicine and material science is domain conceptual term entities, such as DNA, mRNA, ZnO, etc. These noun entities contain neither words with semantic information nor some common roots or affixes, so they are relatively isolated and easy to identify in the text. But the entities in AI and computer science literature are mainly phrasal entities related to the research content, such arxiv 9 Fig. 6. Comparison of performance improvement (relative to baseline) on different scientific literature datasets.
as Task, Method, Process, etc. They contain words with semantic information like "understanding", "combine", "split into", etc., which have higher semantic interweaving with other words in the text, thus brings difficulty for extraction.
The proposed MRC-I2DP model exploits the MRC task form and a gated interactive attention mechanism according to the norms of human reading comprehension. Enhancing the contextual understanding and the overall attention of the target entities, and to some extent, also reduces their semantic interweaving with other words. Therefore, both the above qualitative analysis and quantitative experimental results show that MRC-I2DP has a high ability to understand the content of the text and the target entities.
In addition, to further verify the model's ability to recognize phrasal entities with high comprehension needs and improve the recall when locating nested entities with dense information. The work constructs Table Ⅵ by using the average entity length, entity nesting rate, and whether the phrasal entities are included in each dataset in Table Ⅱ as independent variables and takes the performance improvement on each baseline as dependent variables. We also draw the corresponding line charts, as shown in Fig. 7 and Fig. 8, respectively, for the observation and analysis of performance improvement on datasets with different average entity lengths and entity nesting rates. (Since it is difficult to carry out quantitative statistics for phrasal entities with high comprehension requirements, we adopted the average entity length as the indicator of the comprehension requirement for the model in this paper. The results of Fig. 1 and Table Ⅱ also show that the average length of these phrasal entities is relatively longer.) Fig. 7 illustrates that the overall trend in both scientific and public domain datasets is that "the longer the average length of the entities (i.e., the higher the comprehension requirement for the model), the greater the extraction performance improvement of the model." Fig. 8 also shows the general trend that "with the increase of entity nesting rate, the improvement of extraction performance also increases." It is further verified from two aspects that the model is powerful to understand and recognize the phrasal entities with higher comprehension needs and improving the recall when locating nested entities.
Besides, no matter in the results of Fig. 7 or Fig. 8, the model is best at dealing with the scientific literature that requires high comprehension ability, which is just the weakness of other existing entity extraction models. It sufficiently demonstrates the advantages of the proposed MRC-I2DP model over many other related works.

D. Ablation Experiments and Discussion
The purpose of ablation experiments is to verify the effectiveness of the two critical algorithms proposed to improve the overall performance of entity extraction in MRC-I2DP, namely interactive attention mechanism, and 2D probability encoding. The interactive attention mechanism is used in the reading stage to enhance the understanding of the arxiv 10 context with target entity type and the overall attention to target entities, to improve the recognition performance of phrasal entities, which require higher comprehension in the scientific literature. The 2D probability encoding module is mainly used in answer (i.e., target entity) positioning, aiming to fully and accurately extract the entities (especially those nested entities with intensive information), and further improve the quality and quantity of the extracted results.
Firstly, we carried out the ablation experiment of the interactive attention mechanism, removed the computing parts related to the interactive attention in two Pointer modules and 2DP-Encoder module, and modified the rest parts as same as Li et al. [15] do by concatenating text and question representations and putting them through a fully connected layer. Then re-trained and re-evaluated all the datasets. Secondly, we implemented the ablation of 2D probability encoding, removed the 2DP-Encoder module with the corresponding TaLU function and mask mechanism, and retained only two Pointer modules for generating the starting and ending position distributions as Zheng et al. [16] do to match the two 1D-distribution of start and end positions. Then conducted re-training and evaluating.
The ablation results show in Table Ⅶ, which display that the removal of the interactive attention mechanism results in a relative F1 decrease of 0.3% ~ 3.5% in entity extraction performance on each dataset, and the deletion of 2D probability encoding also relatively reduces the performance by 0.7% ~ 4.4%. As a whole, it proves the effectiveness of the two critical algorithms proposed.
For further exploring the function of interactive attention mechanism in enhancing model's ability of comprehension and phrasal entity recognition in scientific literature, in this section, we also take the average entity length as a degree quantification of the model's comprehension requirement. We construct the performance degradation line chart under the ablation of interactive attention mechanism, as shown in Fig. 9. The result exhibits a general trend that "the longer the average entity length (i.e., the higher the requirement for comprehension), the more the performance decreases." as the proposed interactive attention mechanism is more consistent with human reading comprehension. Firstly, it automatically adjusts the importance of each text part to the answers through an adaptive gating matrix, which is equivalent to the fact that we always decide which paragraphs are more important related to the question and then focus on them when reading. Secondly, the further interaction of entity type, contextual information, and text content is equivalent to our combination of question and context to deeply understand the relevant content, which is greatly helpful to the search of target entities. The ablation result proves the significance and effectiveness of the interactive attention mechanism in context understanding and phrasal entity extraction of scientific literature.
Besides, to further analyze the effectiveness of 2D probability encoding for answer positioning and nested entity extraction, in this section, we construct a performance degradation line chart corresponding to different entity nesting rates with the ablation of 2D probability encoding, as shown in  Fig. 10. The result delivers an overall trend of "the higher the nesting rate of entities, the more performance degradation." The reason that, compared with the positioning method used in related works (i.e., to generate a unique 1D entity range distribution for labeling or to generate and match two independent starting and ending position distributions), the proposed 2D probability encoding has an enhanced ability to detect all the entity boundaries. Each element in the generated coordinate matrix uniquely determines the start-end position pair of an entity, which is of no mismatch error. Therefore, it can effectively deal with some nested entities that contain arxiv 11 Fig. 11. Visualization of the interactive attention matrix. intensive information and improve the extraction recall. So the higher the entity nesting rate, the greater the performance degradation. The ablation result proves the significance and effectiveness of 2D probability encoding in answer (entity) positioning and nested entity extraction.

E. Visualization and Analysis
To show the performance of the MRC-I2DP model more clearly and intuitively, we selected one sample in preprocessed GENIA dataset in this section, of which the text sentence is "PEBP2 alpha A1 and alpha B1 enhanced the expression of the GM-CSF promoter." and the target entity type is Protein. The designed query sentence corresponding to this entity type is "protein nitrogenous organic compounds body tissues muscle hair collagen enzymes antibodies." Firstly, the visualized interactive attention matrix in MRC-I2DP shows in Fig. 11. It illustrates that the word "PEBP2" has the highest attention score with the word "protein" in the matrix, which conforms to that PEBP2 is a protein. In addition, the attention between "alpha A1", "alpha B1", and "antibodies" and that between "expression", "body", and "muscle" are also high, better in line with the biomedical knowledge. Therefore, the model's ability to understand the context and the entity type is verified.
Secondly, we visualized start and end position distributions generated by the Pointer module for target entities and the 2D probability distribution generated by 2DP-Encoder. As shown in Fig. 12, the start position distribution s provides four words in the head of predicted protein entities, namely "PE", "alpha", "alpha", and "GM". The end position distribution e also delivers four tail words: "2", "1", "1", and "F". For most of the relevant models, only the results of start and end position distribution (i.e., s and e, respectively) are used for matching the range of target entities, so that in this case, just "PEBP2", "alpha A1", "alpha B1" and "GM-CSF" can be obtained. However, the output distribution m of 2D probability encoding proposed in this article can capture five targets, including the above four entities and an officially labeled nested entity "PEBP2 alpha A1" with the type of Protein. The visualization result demonstrates the effectiveness of the proposed 2D probability encoding for answer positioning and nested entity extraction.

V. CONCLUSION
In this paper, we proposed a new high-performance model named MRC-I2DP for entity extraction. Through the MRC task form, equipped with the gated interactive attention mechanism and 2D prob-encoding, the model enhanced comprehension of the context and the target entities and improved the ability to capture the answers fully and accurately. Therefore, it effectively alleviates the problem of related models with poor behavior in extracting the phrasal entities in scientific literature and information-intensive nested entities.
The comparative experiments on both scientific and public domain datasets show that the MRC-I2DP outperforms most related representative and SOTA works. It achieves a significant F1 boost of up to 10.4% compared with baseline models, showing higher extraction ability and generalization. Ablation experiments and visualization further confirm that the interactive attention mechanism and 2D prob-encoding can effectively handle the phrasal entities with high comprehension needs and information-intensive nested entities, thus improving the overall extraction quality and quantity.
However, we have only validated the proposed MRC-I2DP model on English texts, so the subsequent work is trying to extract entities from some related Chinese texts to verify its universality.