{'fraction_non_alphanumeric': 0.030469148393676697, 'fraction_numerical': 0.011346251912289648, 'mean_word_length': 4.997196738022426, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Structural block driven - enhanced convolutional neural representation for relation extraction

In this paper, we propose a novel lightweight relation extraction approach of structural block driven - convolutional neural learning. Specifically, we detect the essential sequential tokens associated with entities through dependency analysis, named as a structural block, and only encode the block on a block-wise and an inter-block-wise representation, utilizing multi-scale CNNs. This is to 1) eliminate the noisy from irrelevant part of a sentence; meanwhile 2) enhance the relevant block representation with both block-wise and inter-block-wise semantically enriched representation. Our method has the advantage of being independent of long sentence context since we only encode the sequential tokens within a block boundary. Experiments on two datasets i.e., SemEval2010 and KBP37, demonstrate the significant advantages of our method. In particular, we achieve the new state-of-the-art performance on the KBP37 dataset; and comparable performance with the state-of-the-art on the SemEval2010 dataset.


Introduction
Relation extraction (RE) is an essential task in natural language processing (NLP) to extract the relation between two entities for some given context. In the past years, numerous massive scale knowledge bases (KBs), including DBpedia [1],YAGO [2], and Freebase [3], have been constructed and being broadly utilized in many NLP tasks and applications, such as question answering and web search.
These knowledge bases mainly consist of relational facts with some format, e.g., (Google, founder, Larry Page).
Though existing KBs contain a large number of facts, they are not close to encompassing the vast number of facts embedded in plain text. RE is a method of automatically extracting hidden relational facts from plain text to supplement the KBs.
The relation extraction (RE) tasks are considered into two steps, relation detection, and relation classification. Specifically, the first step is to detect candidate relation mentions in sentences involving pairs of entities, and the second is to classify these relation mentions into predefined categories [4,5,6].
In this paper, we focus on the latter task.
Non-relation instances can be managed as a normal relation class. On the other hand, RE comes with a vastly unbalanced dataset where the quantity of non-relation instances far surpasses relation instances, making this RE task more challenging but more pragmatic than relation classification.
In past decades, most of the work in RE has been dominated by two approaches which can be differentiated by the essence of the relation description: kernel-based approaches [7,8,9,10,11,12,13,14], and feature-based approaches [15,16,13,17]. There is a common goal in these approaches which is to leverage a substantial body of knowledge resources and linguistic analysis in order to map relation mentions into some rich representation. The purpose of the rich representation is that it can be utilized by some statistical classifiers, for example, maximum entropy [18,19] or support vector machines (SVM) [20,21].
The pipeline of linguistic analysis consists of several manually designed steps for example, tokenization, chunking, part of speech tagging, parsing and name tagging, which are often executed by the existing NLP module. Due to the knowledge founded by the NLP research community, these approaches enable the RE module to inherit knowledge from the pre-processing tasks. For example, the indicated tasks in the pipeline above are well known to be subject to significant levels of error when applied to out-of-domain dataset [22,23,24], triggering the RE module to collapse. Thus, our main aim is to propose a novel RE model which reduces the complexity of the feature engineering task, reduce the error propagation rate and improve the performance in the RE task.
In this paper, we propose a novel structural block -driven convolutional neural representation for RE. To be specific, we detect the essential spans associated with entities through dependency relation analysis, by obtaining the parent, siblings, and children nodes of entities. These are ranked into selective sequential tokens in the same order as they appear in the text. We enhance the selective sequential tokens by enriching them with semantic tags (semantic role and part-of-speech tags), all of which is encoded with multi-scale CNNs.
Furthermore, we add two more inter-block representations with one subtract layer of the block representation subtracted by the two entity representations, and one multiply layer of the two entity representations. Then, we concatenate the block-wise and inter-block wise representations to infer the relation. As a result, the encoding of a selective part of a sentence and the enhanced encoding of the block leads to an improvement in both performance and efficiency. We achieved a new state-of-the-art performance in the KBP37 dataset with an F1 of 60.9, and a comparable result in the SemEval dataset with an F1 of 81.1.

Literature Survey
Over recent years, many approaches have been proposed for relation extraction and classification. Most related work are based on applying NLP system or pattern matching to derive lexical attributes. In general, pattern matching is the base for traditional relation classification task [25,26] which can be categorized into kernel-based approaches [9,10] and feature-based approaches [27,16].
The preceding category depends on manually designed patterns and so its time consuming as well as requiring the need the input from experts. Consequently, data sparsity is a challenge facing the latter approaches. Furthermore, extra tools are required for these methods to derive linguistic features.
Distant supervision [28,29,30,31,32] came into much recognition since 2009 to address the challenge of pattern design, and also because of the scarcity of manuially annotated data. This kind of approach integrates knowledge graphs and textual datasets, where the knowledge graph is utilized to automatically identify patterns from the textual dataset.
Our approach is inspired by neural models that learn features automatically, e.g., Collobert et al [33]. Currently, deep learning is [34,35] very widely applied to learn the underlying feature automatically, so the remainder of the literature survey will cover such approaches. For example, Lin [36] proposed a sentence-level attention mechanism to alleviate the wrong labeling problem, expecting to reduce the weight of the instances of noise. They employ a CNNs encoder on multiple sentences with selective attention on expected correctly labeled sentences, by adding a learn-able weight to each sentence. Zhang et. al [37] proposed a model that in the first stage amalgamates the Long Short-Term Memory (LSTM) sequential approach with the type of entity position-aware attention and shows improvement in relation extraction. In the later stage, TA-CRED (106,264 instances), a large supervised relation extraction dataset was attained from crowdsourcing and focused towards TAC KBP relations. This combination of an effective model with high quality supervised data yields superior relation extraction performance. The proposed model outperforms all the neural-based baselines. Culotta et. al [38] proposed a probabilistic extraction model that yields mutual advantage to both "bottom-up" and "top-down" relation extraction. This work demonstrates that amalgamating the relation extraction with pattern discovery improves the performance of each task.
Zeng et. al [39,40,41] used a deep neural network to extract sentence and lexical level features. The proposed architecture takes the input (all the word tokens) without complex pre-processing. Primarily, all the word tokens are converted to vectors by using word embedding. Furthermore, lexical level based features are extracted according to the stated nouns. Meanwhile, the convolutional model is used to learn the sentence level features. The final extracted feature vector is formed by combining these two-level features. In the end, the obtained features are given to the softmax classifier to predict the association between two marked nouns. The obtained results show that the proposed model outperforms the baselines.
One interesting work by Nguyen et. al [42] used CNNs for relation extraction that learns features from the sentence automatically and hence reduces the dependencies on external resources and toolkits. The proposed architecture takes benefit of various window sizes for pre-trained word embedding and filter on a non-static architecture as an initializer in order to enhance performance.
The relation extraction issues because of unbalanced data have been highlighted in this work. Results shown improvement, not only over baselines for relation extraction but also relation classification models. Liu et. al [43] explored how dependency information can be used. Firstly, the newly termed augmented dependency path (ADP) model is proposed, which comprises the shortest dependency path among the two subtrees and entities joined to the shortest path.
In order to explore the semantic relation behind the ADP architecture, they proposed dependency-based neural networks: CNNs to apprehend the most essential features on the shortest path, and a recursive neural network (RNN) is constructed to model the subtrees.
Huang et. al [44] proposed an attention-based CNNs for the relation classification task. The proposed architecture makes full use of word embedding, position embedding and part-of-speech tag embedding information. which part of the sentence is essential and influential w.r.t the two entities of interest, which is determined by a word level attention approach. This model allows learning of some essential features from labeled data, thus removing the dependency on exterior knowledge, for example, the plain dependency structure. The model was tested on the SemEval-2010 Task 8 benchmark dataset and outperforms all the state-of-art neural network models. Furthermore, the model can obtain this performance with minimum feature engineering.
Zhang et. al [45] tried to address the lack of ability to learn temporal features in CNNs, and focussing mainly on long-distance dependencies among nominal pairs. They proposed a general architecture based on recurrent neural networks (RNN's) and contrasted it with CNNs-based approach. A new dataset was introduced which is the refined version of MIMLRE [28]. Experimental results on two datasets demonstrated that RNN's-based approach can enhance the performance of relation classification, and, in particular, is proficeint at learning long distance relationships.

Technical Background
We introduce the essential technical background of our model, including word embedding, CNNs, and semantic parsing. In particular, word embedding initializes the vectors of words; CNNs are the representation models that we employ to encode the text; and semantic parsing is the NLP approach we adopted.

Multi-scale CNNs
CNNs was originally proposed for computer vision but has subsequently been used for text classification [46] where it has shown high performance [47,48] which is superior to traditional NLP based methods.
CNNs is first utilized in a sentence-level classification by Kim et al. [49] where they demonstrated improvement on NLP classification tasks.
Multi-scale CNNs have been demonstrated successful [50] where they employ multi-scale CNNs with different kernel sizes to overcome the drawback of the simple convolutional kernel with fixed window size over encoded semantics of documents, as shown in Figure 1. The reason underlying the design is that determining a fixed window size using a simple convolutional kernel is demanding since small window normally requires deeper networks to gain critical  [50] information while large window sizes lead to loss of local information. Hence, multi-scale CNNs with multiple window sizes are used to represent the comprehensive contextual information of the text.
When we employ this structure, the last Dense and predict layer will be popped out, resulting in a CNN representation, which can be adopted to encode texts of various types, followed by some new neural encoding scheme.

Word Embedding
Word embedding is a very prominent representation of document vocabulary. This allows capturing the context of some given word in a document, associations with other words, syntactic and semantic similarity, etc. Technically speaking, there is a mapping of words into vectors containing real number by utilizing the dimension reduction, probabilistic model, or neural networks on some word co-occurrence matrix. This is a kind of feature learning method and language model as well. Word embedding is a kind of process to execute the mapping utilizing neural networks.
Word embedding. Currently, there are several widely-used word embed-ding approaches for example, Glove (Stanford) 2 , word2vec (Google) 3 , and fastest (Facebook) 4 . In this work, Glove word embedding is used. Glove 5 is type of unsupervised learning model for getting vector representation of words.
Tag Embedding. Besides, we employ one-hot encoding for POS and dependency tokens in our model, since they are not typical words or terms. Particularly, we encode a size of 24 dimensions embedding for POS labels (24 POS tags in total); and 41 dimensions embedding for dependency labels (41 dependency tags in total) in our model.

Dependency Parsing
Dependency parsing 6 is the way to investigate the framework of the sentence, forming an association between headwords and also those words which change the heads. The following Figure 2 explains the dependency style investigation utilizing the traditional graphical approach which is accepted in the community of dependency parsing. Here it is important to note that the lack of nodes analogous to lexical categories or phrasal constituents in some dependency parse; the inner framework of the dependency parse includes simply the directed associations among lexical components in the sentences. Such associations directly encode hidden complex phrase structure parses to essential information.
For example, arguments of a given verb prefer are directly connected in the dependency structure, although the relation to the main verb is far away in the phrase tree framework.

POS Tags
POS 9 tags are very essential for constructing parse tree which are utilized in constructing named entities recognitions (NERs) and extract semantic connections among words. POS is also useful for constructing lemmatizers which minimize words to their root form. POS tagging is the way to highlight a word into the documents dataset to an associated POS tag that is based on some context in which then the word is utilized.

Proposed Method
We propose a novel approach where the entity oriented structural block is detected, then the semantic representation for the block is encoded utilizing multi-scale CNNs, which is further enriched with inter-block representation.
We first introduce the structural block detection in section 4.1, followed by the enriched semantic encoding using CNNs in section 4.2.

Structural Block Detection
We detect the block of coherent tokens for entities in a text. This design is to find directly relevant sequential tokens, whilst retaining their local integrity.
First, we obtain the candidate entities, e.g., in the cases of SemEval and KBP37 dataset, they explicitly give two entities. Next, we detect their structurally related tokens to restore their coherent structural semantics. We achieve this by building up a dependency tree for each sentence with dependency relations between tokens and find the parent, siblings and children nodes as a single block for each entity e i , which is defined as Eq. 1.
where t j is a token from the entity with its rank position j in text; head(t j ) basically refers to the relation name (mostly verbs) while siblings(t j ) refers to those tokens that share the same relation name; and the children(t j ) indicates those tokens that depend on t j . A single block literally covers all the structurally related tokens in terms of a single entity. Nevertheless, we provide an alternative version without including children, i.e., single block(e i ) = t j ∈ei (t j + head(t j ) + siblings(t j )). This version empirically leads to comparable or better performance than the first version, the comparison of which is detailed in section 5.4.
The selected tokens for a set of entity E is defined as Eq. 2 where all single blocks are aggregated into one block defined as aggreg block(E). All the token indexes that are selected are denoted as J = set( t j ∈aggreg block(E) j), indicating that duplicate j will be removed.
Then, the sequential tokens for the block are defined in Eq. 3.
As the block is locally integrated instead of being integrated through the whole sentence, we additionally concatenate its semantic role role(t) and the part-of-speech pos(t) to assist the generalized learning on the selective sequential tokens. The resulting structural block for a sentence is defined as Eq.4, where structural block(E) is the enhanced representation for the selective structural block.
As a result, the relation predicting is defined with softmax function of the probability distribution as below, Exp(x k ) (K is the number of the relation); M is the matrix representation of the relation, s is the enhanced structural block representation; h is the hidden layer; and b is a bias vector in terms of the output.

Modeling
Inspired by [51], we propose an adapted model scheme as shown in Figure   3. The design is to take full advantage of the selective structural blocks with both block and inter-block representation.
For a given sentence and two entities, we first detect the structural block with dependency analysis, resulting in a subset of sequential tokens. We then enrich the semantics by concatenating their semantic role tags and POS tags as demonstrated in Eq. 4. As a result, every single token of the block is represented with three tokens, working together to represent the whole structural block. The semantic role tokens are supposed to give rise to the generalized learning on the selective sequential tokens.
Then, we encode these sequential tokens with multi-scale CNNs, as shown in Figure 1. Specifically, we pop out the softmax layer of the model, resulting in the previous CNN representation layer, to encode the sequential tokens of the entities, dependency labels, POS labels, and structural block.
Besides, we explicitly encode the inter-block representation to gain the connection among the block and two entities. The inter-block representation includes a subtract layer between the block and the two entities, expressed as b − e1 − e2; and a multiply layer between two entities expressed as e1 * e2, assisting the similarity inference between the two entities (as sim(e1, e2) = e1 * e2 |e1|.|e2| ). As a result, we fully connect the structural block representation and the inter-block representation and softmax the final relation classification.

Data
Semantic Evaluation 10 dataset has 10 relations, as listed below. However, because they have order difference, therefore, there is a total of 19 relations, as the "Other" does not apply this rule. The second KBP37 dataset 11 is the revised version of MIML-RE annotation dataset, which was provided by Gabor Angeli et al. [28]. More details about this dataset are presented in table 1 and 2.

Hardware setting
We list the hardware settings used to conduct the experiments. It is importrant to note that we employ ordinary CPU server settings instead of GPUs. Table 3 indicates that our model scheme overall has low setting requirements comparable to most existing servers. We will demonstrate that even on such basic servers the training speed is both fast and efficient.

Evaluation Measures
There are several evaluation measures in machine learning for example, accuracy, precision, recall, f-score, etc. In line with most related work, we employ the F-score to evaluate the performance of our model and use it to compare the performance with other state-of-art neural models.
P recision(P ) = N umber of correctly extracted entity relations T otal number of extracted entity relations Recall(R) = N umber of correctly extracted entity relations Actual number of extracted entity relations (7)

Results and Discussion
As shown in Table 5, our model has superior performance on the KBP37 dataset and comparable performance on the SemEval2010 dataset. The main advantage of our method is that we do not rely on manually determined features while all the other methods adopt a feature set to some degree. As we achieve the new state-of-art performance in KBP37 (with an improvement of 2.1%), we found that text context in KBP37 is much longer than in SemEval2010. Consequently, it seems that our method has a superior advantage in long contexts, since the selective structural block strongly reduces the higher levels of noise associated with long contexts. The confusion matrix is shown in Figure 6, where the mappings of label is listed at footnote 12 .
In SemEval2010, we found that the Instrument-Agency has the lowest F1 (below 70%), which is the bottleneck for the overall results. The reason behind this category can be that it has broader coverage of instances for the two types, namely, Instrument and Agency; while as we can observe that Cause-Effect, Member-Collection, etc. have more specific patterns, or smaller coverage of instances. There are some state-of-art models [52,53] whose F-score is slightly higher than our model, but our model take less time to compute than others. Further, our model rely on manual determined features while all the other methods adopt some level of manually selected features. The confusion matrix is shown in Figure 5.
Our method exhibits satisfactory training speed as well.

Conclusion and Future Work
Relation extraction plays an important role in the population of KBs and other NLP tasks. In this paper, we presented a novel relation extraction approach with few features called the structural block -driven convolutional neural model. The design of the model is to 1) eliminate the noise due to irrelevant parts of a sentence and 2) enhance the relevant block representation, by adopting semantic role embedding and concatenating the inter-block representation.
The block is detected by entity oriented dependency analysis, and the enhanced encoding of the block is conducted not only on block-wise representation but with two more layers, one subtract and one multiply layer, based on strong multi-scale convolutional neural models. We validated our model on two datasets, i.e., SemEval2010 and KBP37, where we achieve the new state-of-the-art performance on the KBP37 dataset (60.9 F1) and comparable performance with the state-of-the-art on the Se-mEval2010 dataset (81.1 F1). As the scheme we proposed is extensible, one avenue for future work is to apply the scheme based on other basic encoders, instead of CNNs. For instance, BERT [56] has proved to be highly efficient in multiple NLP tasks. We can replace the CNNs with BERT as an enhanced BERT representation to observe whether it can generate further improvement.

Acknowledgements
This project receive funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 721321".