{'fraction_non_alphanumeric': 0.03474145486415425, 'fraction_numerical': 0.013321647677475899, 'mean_word_length': 5.133304665663299, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Attention-based Representations from Multiple Patterns for Relation Prediction in Knowledge Graphs

Knowledge bases, and their representations in the form of knowledge graphs (KGs), are naturally incomplete. Since scientific and industrial applications have extensively adopted them, there is a high demand for solutions that complete their information. Several recent works tackle this challenge by learning embeddings for entities and relations, then employing them to predict new relations among the entities. Despite their aggrandizement, most of those methods focus only on the local neighbors of a relation to learn the embeddings. As a result, they may fail to capture the KGs' context information by neglecting long-term dependencies and the propagation of entities' semantics. In this manuscript, we propose {\AE}MP (Attention-based Embeddings from Multiple Patterns), a novel model for learning contextualized representations by: (i) acquiring entities' context information through an attention-enhanced message-passing scheme, which captures the entities' local semantics while focusing on different aspects of their neighborhood; and (ii) capturing the semantic context, by leveraging the paths and their relationships between entities. Our empirical findings draw insights into how attention mechanisms can improve entities' context representation and how combining entities and semantic path contexts improves the general representation of entities and the relation predictions. Experimental results on several large and small knowledge graph benchmarks show that {\AE}MP either outperforms or competes with state-of-the-art relation prediction methods.


Introduction
Humankind's knowledge processed by machines can be posed as sets of information (i.e., facts, beliefs, and rules) circa entities. By connecting these sets of information and their relationships, we can build a network of knowledge. Knowledge bases (KBs) play a fundamental role in compiling this network of knowledge to structure and store information, properties, and relationships.
Existing compiled KBs [1,2] have succeeded as core resources on language and knowledge-related tasks such as question answering [3], information retrieval [4], recommender systems [5], and even commonsense reasoning [6], among others. Typically, the information in these bases are organized in the form of triples (head, relation, tail), in which head and tail are entities (also called subject and object, respectively) and relation (also called predicate) is the relationship between both entities that qualifies the triple semantics. KBs are commonly represented as knowledge graphs (KG), which are labeled multi-digraph whose nodes are the entities (head and tail), the edges express relationships between entities, and the edges' labels define the semantics of each relationship (relation).
Despite the success of KGs as core resources for a large plethora of tasks, realworld KGs usually experience incompleteness and noisy information. They are strictly dependable on new information acquisition, which is often hard to obtain and noisy prone once the information continuously evolves over time [7]. Thus, an essential need regarding KGs arises: the existence of automatic methods to complete their information.
This capacity involves solving the prediction of missing relations among entities in the KG. Such a task is either framed as i) link or entity prediction, when the target is to find a missing entity, given the relation and the other entity ((head, relation, ?), for example); or ii) relation prediction when the target is to find a missing relation ((head, ?, tail)). Often, KGs suffer from completing missing facts given existing ones (relation prediction), differently from question-answering-related tasks, which already have a pair of entity-relationship aiming to find the other correspondent entity (entity prediction) [8,9]. Consequently, targeting completing missing facts in KGs, in this Orange arrows indicate the semantic paths between the head and tail, each one taking a distinct path (BornIn-FriendOf and Play-EnemyWith-Play). Green, red, and blue-shaded areas represent the global, local, and random contexts, respectively, that may be considered by each attention mechanism of AEMP. manuscript, we follow the formalization of the relation prediction task 1 .
Several previous works have proposed techniques to predict and infer new relationships targeting completing knowledge graphs. Such methods can be roughly divided into two categories [12]: i) distributional [13,14,15,16,17] and ii) symbolic methods [18,19]. The former relies on encoding the interactions between entities' neighborhoods into low-dimensional dense vectors, i.e., embeddings vectors, which allow for new relationships between entities to be predicted from interactions of the embeddings [20]. Differently, the later [18,19] aim at predicting new relations from the observed examples in the KG by extracting logical patterns in the form of semantic paths between entities [20]. Methods grounded in the symbolic paradigm can reason over complex relational paths due to their rule-based grounding. However, they are well-known for suffering from scalability issues, which limit their rule-inference search space. In contrast, the embedding-based methods scale, but they can only learn local structures, requiring further enhancements for more generality. Section 2 provides more details on previous work focused on completing KGs.
In this manuscript, we address the relation prediction challenge to complete knowledge bases designing an embedding-based model called AEMP (Attention-based Embeddings from Multiple Patterns). AEMP aims at the major advantages of each paradigm: efficient embeddings learning from the distributional paradigm and capturing generality with semantic paths from the symbolic paradigm. To that, AEMP addresses learning contextualized representations from the combination of entities' context and semantic paths. The key insights of AEMP are based on the perception that multiple patterns exist to relate entities. Like a word in a sentence, entities' local neighborhood evidences their structure and properties, while semantic paths relate to long-term dependencies.
This contextual information's unification enhances information to predict new relations among existing entities in knowledge graphs. We conduct an extensive evaluation of AEMP on the main KGs benchmarks (WN18 [13], WN18RR [21], FB15k [13], and FB15k-237 [22]). Furthermore, we experiment with two small KGs [23] (UMLS and Kinship) to assess AEMP when only limited information is available. We compare AEMP's efficiency against state-of-the-art models in the relation prediction task. The results show that AEMP outperforms the compared methods in five (WN18, WN18RR, FB15k, UMLS, and Kinship) out of the six datasets and demonstrates competitive results on the sixth benchmark. Also, we conduct several ablation studies, where we assess the importance of each pattern observed by the model, namely, the semantic paths and their length, the number of entities context hops on overall performance, and the learning capabilities enabled by each attention mechanism and their combinations. All the results are presented and described in Section 4.
The main contributions of this manuscript are: Red-shaded points out each hop considered within the local attention mechanisms. Green-shaded expresses the iterations associated with the global attention mechanism. Blue-shaded indicates the randomly selected relationships used within random attention. Light and dark shades of the same color illustrate different neighborhood representations of entity h, which are captured and combined by the color-associated mechanism forming the entity m mechanism h . i) An attention-enhanced message-passing scheme for learning contextualized entities representations; ii) Identification of semantic paths between a pair of entities and the combination of the paths' representations into a single general semantic path representation; iii) Combination of contextualized entities representations and semantic path representations for relation prediction; and iv) Extensive experimental evaluation on several knowledge graphs benchmarks showing the proposed solutions either outperforms or competes with existing state-of-the-art methods.

Related work
Most current distributional methods for learning KGs induce entity embeddings based only on their direct neighbors [13,14,21], neglecting that the triples where entities appear are part of a much richer structure (the KG itself). In the following, we revisit methods with similar motivation as ours: entities are likely to be better represented when contextual information is incorporated into the learning process; hence, one may reach better predictive results to complete the KGs. We also review some recent approaches that rely on attention mechanisms over knowledge graphs representation. Finally, as AEMP incorporates the message-passing mechanism, commonly used in graph neural networks, we also bring attention to some of those methods in this section.
Entities Context. Luo et al. [24] pioneered the generation of context-dependent entity embeddings by learning contextual connectivity patterns from 1-hop neighborhood of entities. Later, Oh et al. [25] proposed context-aware embeddings by jointly learning from an entity and its multi-hop neighborhood. Both approaches only adopt the entities' neighbors' representations as context, neglecting the semantics of the relationships between neighbors. In contrast, we build AEMP upon Wang et al. [26], which addressed entities' local context from a message-passing scheme to learn entities k-hop neighborhood based on the aggregation of graph's edges. However, although the message-passing scheme holds the potential to acquire entities' local structures, [26] equally weights the neighbors within the aggregation process. Thus, in this paper, we also benefit from the message-passing scheme, yet we adopt an attention mechanism to focus on different aspects of the passed messages. With that, we weight the importance of each fact in the neighborhood for the final entity representation.
Relational Paths. Several of previous works count with semantic composition among entities to add contextual information to KG's embeddings, either in the form of constraints or as sequence of relations [27,28,29,30,31,32]. PTransE [27] extends the use of translation-based embeddings mechanisms [13] by including multiple-step relation paths between two entities. Gu et al. [28] proposes additive and multiplicative compositions over relations while [33] leverages recurrent neural networks to consider relational paths of entities. Aiming at complex reasoning to populate KBs from texts, [34] and [29] incorporate multiple paths built not only upon relations but also with other entities.
More recently, Guo et al. [31] also targeted at learning from relational paths by using a recurrent neural network with residual connections. AEMP also builds paths motivated by the semantic enrichment that one entity may provide to the others. However, we focus on message-passing schemas leveraged by different attention patterns to let the model find out during the learning which elements are more relevant to the embeddings and, consequently, the relation prediction task.
Attention-based Embeddings. GAT [35] employs the concept of attention to learn representations from graph-based data. It computes attention weights over every other node (in the most general formulation) while not considering possible labeled relations between them. Here, besides allowing different views of attention over the nodes and edges, we aim at predicting relations in knowledge graphs. The first attempt to use attention-based mechanisms to predict missing entities in KGs was presented in [10]. To capture distant entities' contributions, they relied on a relation composition mechanism that introduces auxiliary edges between neighbors in hops. Also, attention mechanisms are used to encode additional information within the learned representation [9,11].
AEMP, on the other hand, allows for local and global attention mechanisms to potentially focus on close and distant neighbors during the message-passing mechanism. Moreover, we include a random attention mechanism that potentially learns when to attend near and distant neighbors to predict missing relations. Selecting neighbors at random to proceed with the message passing introduces an indiscriminate bias once it bases itself on the sparse aspects of the node's neighborhood.
Graph Neural Networks. Commonly, Graph Neural Networks (GNN) methods follow the propagation-aggregation node-based message-passing paradigm over multi-hop neighbors [36,37,38,39]. AEMP also follows the message-passing schema, yet the messages are passed along the edges to capture semantic relationships expressed on them. The entities' messages are aggregated from the relation representations that connect them. Previous approaches have also explored GNN to complete knowledge graphs, primarily focusing on link/entity prediction tasks. Several of them leverage on the local neighborhood context of entities only [40,41,10,42] while here we explore global and random contexts besides the local neighborhood. There are many ways to explore local information. For instance, Zhang et al. [42] assume there is a hierarchical structure in the local neighborhood of an entity while Schlichtkrull et al. [40] and [43] extend Convolutional Neural Network to knowledge base completion by addressing the local context of nodes. Teru et al. [44] also take advantage of a GNN framework yet assuming an entity-independent semantics to learn to predict relations to handle inductive reasoning problems. They built the learning framework upon several other approaches and offered a set of inductive relation prediction benchmarks to the community. Evolving AEMP to the inductive setting, particularly regarding the semantic paths component, is an appealing direction for the future.

Our Approach: Attention-based Embeddings from Multiple Patterns
In this section we propose AEMP 2 (Attention-based Embeddings from Multiple Patterns). Figure 2 depicts an illustration of how the different components of the proposed architecture interact. Roughly speaking, AEMP includes entities' context enhanced by three attention mechanisms and semantic paths to induce the embeddings of entities. Those embeddings are the input of a softmax function that defines the probability of a relation connects two entities, i.e., a (missing) candidate triple. We start by defining the problem we tackle. Next, we provide AEMP components in details.
where E is the set of nodes that represents the entities, R is the set of edges' labels that represents the relations, and F : E × R × E is the set of triples (h, r, t) that represent logical facts r(h, t). Our goal is, given a pair of entities where h ∈ E and t ∈ E, to predict the existence of a relation r connecting these two entities given by the probability P (r|h, t). The outcome assembles a new (missing) where r ∈ R is the predicted relation, and f is the candidate triple.
The notations used in this section and the rest of this manuscript, together with their descriptions, are presented in Table 1.

Entities Context
Here we describe the attention-enhanced mechanism for learning the joint represen- context. The development of this attention-enhanced procedure is inspired by the advances in attention mechanisms over graphs (GATs) [35], as well as their sparse versions [45,46]. The message-passing scheme resembles the aggregating entities' neighborhood representations proposed in [26].
Equations 1 calculates the message representation of an entity e according to the massage-passing scheme, while Figure 2.(b) illustrates it. The message m i e ∈ R D is the contextual representation of an entity e ∈ E, acording to the context captured as s i r , which is the representation of a relationship r (an edge) between a pair of entities: Both m i e and s i r are the entity message and relationship representation computed in the i-th iteration of the outer loop of the Algorithm 1. Note that the message associated with an entity e is built upon the aggregation of the relationship's representation that connects e to another entity, according to e's neighborhood N (e). The messages of two entities are combined through a pairwise product: Precisely, the entity's message representation m i e ∈ R D calculated within an iteration i is the sum of all current incident relationship representation s i r where r ∈ N (e) (the neighbourhood of the entity e) (Equation 1). The relation representation is also updated iteratively according to Equation 2, where the next relation representation s i+1 r is updated based on a cross-neighbor aggregator operation [26]. The aggregator models the entities' cross matrix, a pairwise product of the entities' message representations Built on top of the message-passing scheme, we define a multi-context attention layer motivated by GAT and the sparse attention mechanism [46]. We extend the GATs' attention mechanism by adopting local and global attention. Moreover, we include random attention within the message-passing scheme iterations to leverage from sparsely arranged possible relationships.
Assume a particular iteration 1 ≤ k ≤ K, where K is the maximum number of iterations according to Algorithm 1. The local attention mechanism focuses on the relationships' messages acquired in each hop visited within k (inner loop of Algorithm 1) as the context message s k resultant of that iteration. The context message allows for the querying message to be informed of its local context structure captured by its neighborhood structural information. Figure 2.(c) illustrates this mechanism and Equations 3 and 4 formalize it: where α local rk is the local attention weight calculated at iteration k when focusing on a relation r, s k is the gathered representation of relations participating at the iteration k, s k r is the representation of a particular relation r at the iteration k, CR r is the set of other relations in the relation r's local context, i.e., those relations sharing a node with r, and {W lα , W ls } ∈ R D × D are arbitrary weights associated with the local attention.
Finally, s local k r is the local relation representation. To calculate it, AEMP first calculates a linear composition of the local attention weights and representation of relations that belong to the context CR r of r, weighted by a learned matrix W sα . After, to finally get where α global rk are the global attention weights, s r is the representation of the relation r detached from any iteration, while s k r is the representation of the relation r computed within the iteration k, {W gα W gs } ∈ R D × D , W g are learned according to the global attention, and C I is the set of relations taken into account during the iteration k.
Note that the local attention has the point of view of representations computed at a iteration k, namely s k and s k r while the global attention also focuses on s k r but aligns it with the representation of the relation detached from a specific iteration, namely, s r .
Finally, the random attention, illustrated in Figure 2. The random attention mechanism is formalized as follows: where α random rk are the random attention weights, s is a representation captured from any relation, s k r is the representation of the relation r at the iteration k, CR r is the context of a relation, {W rα W rs } ∈ R D × D , W r are learned to weigh the random attention weights.
computed over similar conditions of the local message.
Finally, the message find out by the random attention mechanism is defined by the following equation, with similar definitions as above.
Finally, to provide the final embedding of the head and tail pair M (h,t) , we concatenate the head and tail final messages: where m AT T h and m AT T t are the final representation messages from entities head and tail, respectively.

Semantic Path Context
Next, we describe the capture of semantic paths (Figure 2.(a)) and how they are aimed as context information within the learning process. The aforementioned attentionbased approaches provide essential information for learning entities' local context.  However, it may fail at capturing long semantic paths, i.e., long sequences of relations between entities. As an example, the semantic path ( Figure 1 yellow-shaded arrows) between PatrickStewart and IanMcKellen entities is overlooked, since it is longer than the entities context hops. To address this issue, AEMP introduces representations learned from the semantic paths between entities when learning the embeddings to predict a missing relation.
Algorithm 2 describes the procedure for capturing the semantic paths and Figure 3 illustrates the mechanism. The procedure first identifies m semantic paths P(h, t) = where W p denotes the linear transformation matrix weights, and denotes the concatenation operation.
Algorithm 2: Learning semantic paths representation

Training Objective
To predict relations over KGs, we rely on a loss function based on the distribution probability of a relation over a head and tail pair [26]. Like so, we take the probability distribution computed by a softmax function from the addition of both entities context M (h,t) and semantic path context P (h,t) representations: We target at minimizing the cross-entropy loss between the predicted probability of a training fact T and its ground-truth: r log(P (r|h, t; Ω)).

Experiments
We conduct experiments on real-world KGs largely used in the literature and report the results compared with several previous methods that have also focused on the relation prediction task. Furthermore, we conduct three ablation studies, where we draw insights over AEMP and its variations.  [48]. We used the originally provided data splits for WN18, WN18RR, FB15k, and FB15k-237, and for UMLS and Kinship, we use the data splits provided by [10].
Baselines. When experimenting with WN18, WN18RR, FB15k, and FB15k-237, we compare AEMP, and its variations, with six models that have results also targetting at the relation prediction task: TransE [13], ComplEx [14], SimplE [49], RotatE [15], QuatE [16] are the models representing the state-of-the-art in embedding-based models, while PathCon [26] is the state-of-the-art closer to our model, using entities context and relational path as features in the learning process. We reused the results reported in [26]. We trained from scratch UMLS, and Kinship with PathCon [26] as this is the most similar method to AEMP and the SOTA in the relation prediction task.
Implementation details. We implemented AEMP 3 using PyTorch [ and FB15k-237 benchmarks, we employed two-hops entities' context and semantic path length up to two, due to hardware limitations and the size of each benchmark.
Evaluation protocol. We evaluate all models on their capabilities of completing missing

Results
The empirical results are reported in Table 2 and Table 3 on the large datasets (WN18, WN18RR, FB15k, and FB15k-237), and in Table 4 on the small datasets (UMLS, and Kinship). L, G, and R stand for AEMP relying on local, global, and random attention mechanisms only, respectively. When there is more than one mechanism taking place, we add two or more of those letters. AEMP achieves competitive results, outperforming the other methods on all metrics for WN18, WN18RR, UMLS, and Kinship datasets and In general, when AEMP uses the global attention only, it achieves the best overall results reaching the best or the second-best cases in seven cases for the large datasets.
Following, we see that the random attention mechanism is in six times in the first two positions (either it is the best or the second-best result), followed by the combination of local and global attention mechanisms and the random and global attention mechanisms

Ablation Studies
To better assess the capabilities of AEMP we conduct three ablation studies. As such, we (i) evaluate the benefits of using semantic paths' representations towards the capacity of AEMP to predict relations, and (ii) investigate the influence of the number of hops and the number of context neighbors, towards the predictions' results. The results of the first ablation studies are illustrated in Figure 4, and details of used parametrization are shown in Appendix C. Finally, the third (iii) ablation study, illustrated in Figure 5, inspects closely the relation between correct predictions made by AEMP and its variations and the relations' frequency within the KG. The parameters' settings in the fourth study are the same as those used in the previous section.
In the first study (Figure 4a), we assess the top 1 hit ratio (Hit@1) on WN18RR, varying each attention mechanism and their combination alongside either enabling or disabling the representation of semantic paths. Considering the variations that implement only a subset of the attention mechanisms, AEMP achieves its best performance when it is able to combine both entities' context representation with the representation of the   indicates that the attention-enhanced message-passing scheme might be able to not only represents entities' local neighborhood but also to learn longer sequences of relationships, i.e., the semantic paths between the head and tail entities.
In the second study (Figure 4b), we assess the top 1 hit ratio on WN18RR regarding different numbers of entities' context hops and semantic path length. AEMP (circle symbol) achieves the overall better results using two contexts hops. In comparison, the subset variations achieve overall better results using the maximum numbers of entities' context hops and semantic path length. Those results reinforce the previous study, suggesting that AEMP with its attention-enhanced message-passing scheme have generalizations capabilities, capturing contextual semantics with less context information.
The third study's goal is to investigate the learning capabilities enabled by each attention mechanism and their combinations. To that, we analyze the Hit@1 performance of AEMP and its variations regarding each existing relation within the WN18RR dataset. Figure   better results, achieving good higher hit ratios on most relations.

Conclusion
In this paper, we devised AEMP, a novel learning method that learns contextualized representations from the combination of distinct views of entities context and semantic path context. Our results show that AEMP has the potential to outperform state-of-the-art models in the relation prediction task. The empirical analysis indicates that the proposed attention-enhanced message-passing scheme can represent not only the entities and their context but also the semantic path context. As future works, we point out three research questions: (i) can the proposed message-passing scheme be generalized to represent even longer sequences of relations, replacing the representation of semantic paths?; (ii) how AEMP compares with pure symbolic approaches according to their reasoning skills?; and (iii) may other design alternatives (e.g., recurrent networks or transformers) better represent longer semantic paths? In this section, we conduct two more studies on AEMP. The first evaluates the scalability of AEMP while the second evaluates the use of a sparse or dense representation for the semantic path representation. The second extended study (Table C. To do so, we replace the semantic paths capturing process described in 3.2 for a recurrent neural network. We measure the MRR and Hit@1 metrics on the six used datasets and insert the results in Table C.8. 0.9757 ± 0.0004 0.9583 ± 0.0007 0.9359 ± 0.0091 0.8922 ± 0.0120 0.8983 ± 0.0093 0.8160 ± 0.0175 AEMP (L+G+R) 0.7745 ± 0.0283 0.6841 ± 0.0330 0.9281 ± 0.0216 0.8794 ± 0.0376 0.9385 ± 0.0043 0.8908 ± 0.0084 Also, to eases comparison with the representation proposed in AEMP, Figure C.7 shows the absolute difference of the best Hit@1 metric between representations of each model on each dataset. In general, both representations presents similar results in all model-dataset pairs. Specifically, the results indicate a slight performance gain using sparse representation for larger dataset (such as WN18 and FB15k), and, on smaller datasets (UMLS and Kinship), the gains tends to be neutral.