{'fraction_non_alphanumeric': 0.03825397717690708, 'fraction_numerical': 0.011242902917533307, 'mean_word_length': 5.078763027507261, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs

Knowledge Graphs (KGs) are ubiquitous structures for information storagein several real-world applications such as web search, e-commerce, social networks, and biology. Querying KGs remains a foundational and challenging problem due to their size and complexity. Promising approaches to tackle this problem include embedding the KG units (e.g., entities and relations) in a Euclidean space such that the query embedding contains the information relevant to its results. These approaches, however, fail to capture the hierarchical nature and semantic information of the entities present in the graph. Additionally, most of these approaches only utilize multi-hop queries (that can be modeled by simple translation operations) to learn embeddings and ignore more complex operations such as intersection and union of simpler queries. To tackle such complex operations, in this paper, we formulate KG representation learning as a self-supervised logical query reasoning problem that utilizes translation, intersection and union queries over KGs. We propose Hyperboloid Embeddings (HypE), a novel self-supervised dynamic reasoning framework, that utilizes positive first-order existential queries on a KG to learn representations of its entities and relations as hyperboloids in a Poincar\'e ball. HypE models the positive first-order queries as geometrical translation, intersection, and union. For the problem of KG reasoning in real-world datasets, the proposed HypE model significantly outperforms the state-of-the art results. We also apply HypE to an anomaly detection task on a popular e-commerce website product taxonomy as well as hierarchically organized web articles and demonstrate significant performance improvements compared to existing baseline methods. Finally, we also visualize the learned HypE embeddings in a Poincar\'e ball to clearly interpret and comprehend the representation space.


INTRODUCTION
Knowledge Graphs (KGs) organize information as a set of entities connected by relations. Positive first-order existential (PFOE) queries such as translation, intersection, and union over these entities aid in effective information extraction from massive data (see Figure 1 for an example PFOE query). Efficient handling of such queries on KGs is of vital importance in a range of real-world application domains including search engines, dialogue systems, and recommendation models. However, the large size of KGs and high degrees of the nodes therein makes traversal for querying a computationally challenging or, in some cases, even an impossible task [43]. One way to resolve this issue is to learn representations for the KG units (entities and relations) in a latent (generally Euclidean) space such that algebraic or logical operations can be applied to extract relevant entities. Robust representation learning of KG units has several real-world applications including KG information extraction [20], entity classification [46], and anomaly detection [22]. Earlier approaches to representation learning in KGs model entities and relations as vectors in the Euclidean space [7,31,48]. This is suboptimal due to the constant size of a point's answer space which does not capture the variations induced by different queries. Specifically, broad queries (Nike) should intuitively cover a larger region of the answer space compared to specific queries (Nike running shoes for men). In the recently proposed Query2Box model [35], the authors demonstrated the effectiveness of complex geometries (such as hyper-rectangles) with varying offsets that control the size of an answer space according to a query's complexity.

arXiv:2012.13023v3 [cs.LG] 12 May 2021
However, such architectures lack the ability to capture hierarchical information that is prevalent in many KGs. Furthermore, previous representation learning methods in heterogeneous graphs (or KGs) [16,25,26,51] solely focus on one-hop or multi-hop reasoning over relations. Such frameworks enable static and optimized computational graphs, but lead to poor retrieval from complex intersection and union queries. Dynamic computational graphs, which are able to modify their network architecture with a switch mechanism (discussed in Section 3.5) can significantly alleviate this problem.
Although Euclidean spaces have proven to be effective for representation learning in various domains [3], several hierarchical datasets (including graph data) in the fields of network sciences and E-commerce taxonomies demonstrate a latent non-Euclidean anatomy [8]. The introduction of hyperbolic algebraic operations [17] have led to the proliferation of hyperbolic neural networks such as Hyperbolic-GCN (H-GCN) [12] and Hyperbolic Attention (HAT) networks [19]. These frameworks leverage the hyperbolic anatomy of hierarchical datasets and show a significant performance boost compared to their Euclidean counterparts. To the best of our knowledge, there is no existing work that (i) utilizes dynamic computational graphs on the hyperbolic space, (ii) applies complex hyperbolic geometries such as hyperboloids for representation learning. Additionally, the static computational graphs of H-GCN and HAT limit their learning capability to a single problem, generally, multi-hop (translation) reasoning. This severely limits their applicability to representation learning on KGs since translations can only utilize single entities. More complex intersections and unions not only use more entities, but are also more representative of real-world KG queries. While solving union and intersection queries is more challenging, they enable better representation learning [4]. Traversing over the entities in KGs facilitates an intuitive way of constructing a query-reasoning proxy task (refer Section 4.3) that enables representation learning of entities and relations. These representations, in a self-supervised framework, can further provide enriched features in downstream tasks with smaller annotated datasets (such as anomaly detection), thus alleviating the issue of data scarcity. Motivated by the effectiveness of self-supervised learning and the need for non-Euclidean geometries in KGs, we formulate KG representation learning as a self-supervised query reasoning problem. We introduce Hyperboloid Embeddings (HypE), a self-supervised dynamic representation learning framework that utilizes PFOE queries to learn hyperboloid representations of KG units in a (non-Euclidean) Poincaré hyperball. Hyperboloids, unlike vectors in hyperbolic spaces, allow us to use dynamic sizes for KG representations. For e.g., in Figure 2, we can notice that different entities contain different number of children, and, thus learning a static vector representation is suboptimal. Hyperboloids learn an additional spatial parameter, limit (described in Section 3.4), that can model the varying entity sizes. Moreover, the dynamic nature of its computational graphs allows HypE to utilize different network layers to learn different types of operations, namely, translation, intersection, and union; and process varying number of input units depending on the learning operation. Our empirical studies include learning representations from large-scale KGs in e-commerce, web pages (DBPedia), and other widely used Knowledge bases (such as Freebase and NELL995); and evaluating the representations on the downstream task of anomaly detection. The major contributions of this paper are: (1) Formulate the KG representation learning problem as a selfsupervised query reasoning problem to leverage PFOE queries. (2) Introduce Hyperboloid Embeddings (HypE), a self-supervised dynamic representation learning framework that learns hyperboloid representations of KG units in a Poincaré hyperball. This is motivated by the need for non-Euclidean geometries. (3) Perform an extensive set of empirical studies across diverse set of real-world datasets to evaluate the performance of HypE against several state-of-the-art baseline methods on the downstream task of Anomaly Detection. (4) Visualize the HypE embeddings to clearly interpret and comprehend the representation space. The rest of the paper is organized as follows: Section 2 describes the related background. Section 3 formulates the representation learning problem, explains the non-Euclidean algebraic operations, and the proposed HypE model. In Section 4, we describe the real-world datasets, state-of-the-art baselines and performance metrics used to evaluate the HypE model. We demonstrate the performance results along with the visualization of HypE's representations. Finally, Section 5 concludes the paper.

RELATED WORK
In this section, we review different geometries utilized for learning representations and earlier works that are adopted for reasoning over Knowledge Graphs.
Representation Geometries : Previous approaches to representation learning, in the context of KG, aim to learn latent representations for entities and relations. Translational frameworks [7,32,48] model relations using translation between entity pairs. This limits the models to only handle translation-based queries. Graph Query Embedding (GQE) [20] overcame this limitation and provided a technique for leveraging intersection queries as deep sets [49] over different queries. Furthermore, Box Lattices [41], EMQL [37] and Query2Box [35] proved the effectiveness of more complex geometries (hyper-rectangles) for queries. Word2Gauss [42] is a popular NLP technique that learns Gaussian embeddings for words. DNGE [39] utilizes a dynamic network to learn Gaussian embeddings for entities in a graph. These Gaussian representations cannot be intuitively extended to KGs because they are not closed under more complex PFOE queries (intersection or union of Gaussians does not yield a Gaussian). Furthermore, they rely on properties of the Euclidean space to learn representations, which are proven ineffective at capturing the prevalent hierarchical features of a KG [17].
Representation Learning on Graphs : One of the fundamental problems in KG is to aggregate the neighbor information of nodes while learning representations. Node embedding techniques such as Node2Vec [18] and DeepWalk [34] aggregate the neighbors' features by modeling the node's dependence on its neighbors. Cheb-Net [13] uses Chebyshev ploynomials and filters node features in the graph Fourier domain. GCN [23] constrains the parameters of ChebNet to alleviate overfitting and shows improved performance. Graph-BERT [51] and MAGNN [16] provide a self-supervised learning model utilizing the tasks of masking and metapath aggregation, respectively. In another line of research, Miller et al. [28] utilizes non-parametric Bayesian frameworks for link prediction on social networks. Zhu [52] further improved the approach with a maxmargin framework. KGAT [44] is another popular approach that utilizes attention networks over entities and relations with a TransR [24] loss function to learn representations for user recommendation. These methods rely on relational properties and thus are effective in handling translational problems such as multi-hop reasoning. However, they are ineffective at handling more complex PFOE queries such as intersection and union.
Other popular multi-hop graph networks such as Graph Attention Network (GAT) [40] and Graph Recurrent Network (GRN) [36] have previously shown impressive results in reasoning-based QA tasks. However, hyperbolic flavors of these networks, H-GNN [17], H-GCN [11,12] and H-GAT [19] argue that hierarchical datasets follow the anatomy of hyperbolic space and show improved performance over their Euclidean counterparts. Nonetheless, these approaches are still limited by the constant answer space that does not consider the varying fluctuations of complex queries.
Self-supervised learning [14,21,29,47] utilizes large unannotated datasets to learn representations that can be fine-tuned to other tasks that have relatively smaller amount of annotated data. Traversing over the entities in KGs facilitates an intuitive way of constructing a query-reasoning proxy task (refer Section 4.3) that enables representation learning of entities and relations. These representations, in turn, are employed in downstream tasks with scarce datasets such as anomaly detection. The proposed HypE model utilizes a self-supervised learning framework that leverages both simple and complex PFOE queries to learn hyperboloid (with varying limits) representations of KG units in a Poincaré ball to efficiently capture hierarchical information.

PROPOSED FRAMEWORK
In this section, we first provide the standard method of querying knowledge graphs. Then, we set up the problem and describe the details of our model that learns representations of entities and relations from reasoning queries over Knowledge Graphs (KG).

Querying Knowledge Graphs
KGs contain two primary units, namely, entities and relations. Entities are the basic information units that connect to each other by relation units. Heterogeneous graphs [45,50] can be considered as a special case of KGs where the relations serve as hierarchical connections with no inherent information. PFOE queries of translation (t), intersection (∩) and union (∪) serve as the primary means of querying these KGs. Translation queries utilize an entity and a relation to retrieve all entities that are connected to through . An equivalent example of translation query for heterogeneous graphs is to retrieve all children of a node connected by a certain edge type . Intersection and Union operate over multiple entities = { 1 , .., } and correspondingly retrieve the set of all entities that are connected to all ∈ and any ∈ . For heterogeneous graphs, the equivalent is to retrieve nodes connected to all nodes ∈ and any ∈ . An example of PFOE querying is given in Figure 3. The widely studied problem of multi-hop traversal [16] is a more specific case of translation queries, where multiple queries are chained in a series.

Problem Setup
We denote = ( , ) as a set of entities ∈ and relations ∈ : → as Boolean functions that indicate whether a directed relation holds between and . Intersection (∩) and Union (∪) are positive first-order existential (PFOE) operations defined on a set of queries 1 , 2 , ...., ∈ : where ( ∩ , ∩ ) and ( ∪ , ∪ ) is the query space and resultant entity set after the intersection and union operations over query set , respectively 1 . Given a dataset of logical queries over a , the goal, here, is to learn hyperboloid (in a Poincaré ball) representations for its entities ∈ R 2 and relations ∈ R 2 , where is a hyperparameter that defines the dimension of the Poincaré ball.

Manifold Transformation Layer
Hierarchical structures intuitively demonstrate the latent characteristics of a hyperbolic space [17]. Thus, we utilize the Poincaré ball [9] to model our representations. 3.3.1 Transformation from Euclidean Space. The transformation from Euclidean to hyperbolic space (H , H ), given in [17], is defined by the manifold H = { ∈ R : ∥ ∥ < 1} with the Reimannian metric , H , where: (1) E = I being the Euclidean identity metric tensor, and ∥ ∥ is the Euclidean norm of .
is the conformal factor between the Euclidean and hyperbolic metric tensor set to a conventional curvature of -1. Eq. (1) allows us to convert a Euclidean metric to hyperbolic. Thus, the distance between points , ∈ H is derived as: 3.3.2 Gyrovector Spaces. Algebraic operations such as addition and scalar product which are straightforward in the Euclidean space cannot be directly applied in hyperbolic space. Gyrovector spaces allow for the formalization of these operations in hyperbolic space. Ganea et al. [17] provide the gyrovector operations relevant to training neural networks. The gyrovector operations for Poincaré ball of radius are Möbius addition (⊕ ), Möbius subtraction (⊖ ), exponential map (exp ), logarithmic map (log ) and Möbius scalar product (⊙ ).
, ∀ ∈ R, ∈ H Here, denotes assignment operation for Möbius operations. Also, the norm of , can subsume the scaling factor . Hence, in HypE, training can be done with a constant or trainable . We empirically validate this assumption in our experiments (Section 4.4). Figure 4 shows an example of the manifold transformation from Euclidean space to a Poincaré ball of unit radius. HypE extends the operations to handle complex geometries, explained in Section 3.4.

Dynamic Reasoning Framework : HypE
We aim to learn hyperboloid (made of two parallel pairs of arcaligned horocycles) embeddings for all the entities and relations in the KG. An arc-aligned horocyle ( Figure 5(a)) is a partial circle that is parallel to the diameter of a Poincaré ball and orthogonally   The architecture utilizes a switch mechanism to connect/disconnect different layers according to the query operator signal (t,∩,∪). The blue, red and green switches are connected for translation, intersection and union operations, respectively (and disconnected otherwise). The yellow and pink circles depict the center and limit of KG units, respectively. This figure is best viewed in color.
intersects its boundaries at two points. A hyperboloid embedding (see Figure 5(b)) = (cen( ), lim( )) ∈ R 2 is characterized by: where = = denotes strict equivalence and ≤ is element-wise inequality and cen( ), lim( ) ∈ R are center of the hyperboloid and positive border limit (lim( ) ≥ 0) of the enclosing horocycle from the center, respectively. The overview of the architecture is given in Figure  6. From the KG, we derive the following types of directed edge relations to build our dynamic computational graph for learning embeddings.
Distance between hyperboloid and entity point (d): Given a query hyperboloid ∈ R 2 and entity center ∈ R , the distance between them is defined as: where represents the distance of the entity to limits of the hyperboloid and is the distance of the entity from the hyperboloid's border to its center. is a scalar weight (set to 0.5 in our experiments) and ∥ ∥ 1 is the 1-norm of x.

Translation (t):
Each relation ∈ is equipped with a relation embedding = ∈ R 2 . Given an entity embedding ∈ , we model its translation ( ) and distance from the result entities ∈ ⊆ ( ) as follows: This provides us with the translated hyperboloid with a new center and larger limit (lim( ) ≥ 0). A sample operation is illustrated in Figure 7(a).
(.) and exp(.) are element-wise minimum and exponential functions. The new center and limit are calculated by an attention layer [1] over the hyperboloid centers and DeepSets for shrinking the limits, respectively. Figure 7(b) depicts a sample intersection.
Union (∪): Unlike intersection, union operations are not closed under hyperboloids (union of hyperboloids is not a hyperboloid). Hence, the distance of entities from the union query space ( ∪ ) is defined as the minimum distance from any hyperboloid in the union. For a set of hyperboloid embeddings ∪ = { 1 , 2 , 3 , ..., }, union space is given by ∪ and distance from result entities ∪ ∈ ∪ ⊆ by ∪ , where Note that, since union is not closed under hyperboloids it cannot be applied before the other operations. We circumvent this problem by utilizing Disjunctive Normal Form (DNF) transformation [35] on our logical queries. This allows us to push all the union operations to the end of our computational graph, thus maintaining validity for all PFOE queries. An outline of HypE's training procedure is given in Algorithm 1.

Implementation Details
We implemented HypE in Pytorch [33] on two Nvidia V100 GPUs with 16 GB VRAM. For gradient descent, the model is trained using Reimannian Adam optimizer [2] with an initial learning rate of 0.0001 and standard values of 0.9 and 0.999. We utilize ReLU [30] as the activation function. Also, we randomly selected 128 negative samples per positive sample in the training phase to learn better discriminative features. For our empirical studies, we learned hyperboloid embeddings of 2 × 400 dimensions (d=400). Due to the conditionality (i.e., if conditions in Algorithm 1) in our computational graph, we employ a switch mechanism between the network layers [15,27]. The switch mechanism receives an operator signal that defines the operation and accordingly connects/disconnects a layer from the framework. A disconnected switch blocks backpropagation of weight updates to the disconnected layers. This enables a concurrent use of all PFOE queries to update the entity and relation embeddings. For an input query and resultant entities , Algorithm 1 provides the pseudocode of our overall framework to learn representations of entities and relation . The algorithm describes the three main operations, namely, translation (lines 4-7), intersection (lines 8-11), and union (lines 12-15) 2 .

EXPERIMENTAL SETUP
This section describes the experimental setup that analyzes the performance of HypE on various problems. We aim to study the following research questions:

Datasets
We perform our experimental study on the following standard KG and hierarchical graph datasets: (1) FB15k [7] contains knowledge base relation triples and textual mentions of Freebase entity pairs. This dataset contains a large number of simple test triples that can be obtained by inverting the training triples. (4) DBPedia Hierarchical Taxonomy 3 is a subset extracted from Wikipedia snapshot that provides multi-level hierarchical taxonomy over 342,782 articles (leaf-nodes). (5) E-commerce Product Network 4 is a subsampled product taxonomy from an e-commerce platform.
To be consistent with KG terms, for the hierarchical graph datasets (DBPedia and E-commerce) we consider all the intermediate and leaf nodes as entities and the edges between them as relations. Additionally, we consider two variants for encoding edges. First, all edges are considered identical (| | = 1) and second, where all edges are depth-encoded (| | = ), where is maximum depth of the hierarchy ( = 3 for DBPedia and = 4 for E-commerce dataset). For cross-validation and evaluation, we split the graph into three parts: , and in a 75 : 10 : 15 ratio for our experiments. More details of the datasets are given in Table 1.

Baselines
We select our baselines based on the following two criteria: (1) The embedding geometries are closed under the intersection and translation operation, e.g., the translation or intersection of arc-aligned hyperboloids results in an arc-aligned hyperboloid. (2) The baseline can be intuitively extended to all PFOE queries over KG. This is necessary to have a fair comparison with HypE that can leverage all PFOE queries.
We adopt the following state-of-the-art baselines based on geometric diversity and our criterion to compare against HypE: • Graph Query Embedding (GQE) [20]

RQ1: Efficacy of the Query-Search space
To analyze the efficacy of the query space obtained from the HypE model, we compare it against the state-of-the-art baselines on the following reasoning query structures: (1) Single operator queries include multi-level translation (1t, 2t, and 3t) multi-entity intersection (2∩, 3∩) and multi-entity union queries (2∪). 1t, 2t, and 3t denote translation with 1, 2 and 3 consecutive relations, respectively. 2∩ and 2∪ stand for intersection and union over two entities, respectively. 3∩ represents intersection over three entities. (2) Compound queries contain multiple operators chained in series to get the final result. Our experiments analyze ∩ (intersectiontranslation), ∩ (translation-intersection) and ∪ (union-translation). The above queries are illustrated in Figure 8. We extract the ground truth query-entity pairs by traversing the datasets [35]. The models are trained on queries from and validated on . The final evaluation metrics are calculated on . We utilize Euclidean norm and hyperbolic distance (given in Eq. (3)) to measure the distance between query embeddings and its resultant entities in Euclidean and hyperbolic spaces, respectively. The sorted query-entity distances are the ranked results for the given query. Table 3: Performance comparison of HypE (ours) against the baselines on an e-commerce dataset for multi-level translation queries. We assume GQE (E-vector) as a baseline and report relative improvements against that for all the methods. The numbers are in percentages. Best results for each dataset are shown in bold.  Reciprocal Rank (MRR). The metrics are given by: From the results in Table 2, we observe that, on average, HypE outperforms the current baselines in translation, single, and compound operator queries by 15%-67%, 4%-13%, and 7%-28%, respectively. The performance improvement linearly increases with higher query depth (1 < 2 < 3 ). Furthermore, we notice that unique depth encoding (| | = ) outperforms identical depth encoding (| | = 1) by 23% in DBPedia. Because of our subsampling strategy, the E-commerce Product Network is disjoint (i.e., there are several intermediate nodes that do not share children). Hence, the number of intersection queries are extremely low and insufficient for training HypE or baselines. However, we can still train the translation queries, and the results are shown in Table 3, where we report relative performance improvements with respect to the GQE baseline. For the sake of completeness, results on intersection and union queries are given in the Appendix C.

RQ2: Ablation Study
In this section, we empirically analyze the importance of different layers adopted in the HypE model. For this, we experiment with different variations of the center aggregation layer; Average (HypE-Avg), Attention (HypE) (refer to Eq. (6)) and Deepsets (HypE-DS) (refer to Eq. (7)). Furthermore, we test the exclusion of intersection and unions to comprehend their importance in the representation learning process. We adopt two variants of HypE; one trained on only 1t queries (HypE-Avg-1t) and the other trained on all translation queries (HypE-Avg-1,2,3t). Table 4 presents the performance metrics of different variants on the query processing task, averaged across all the datasets including the E-commerce dataset. The results are provided in Table 4.
Firstly, we observe that the exclusion of intersection and union queries results in a significant performance decrease by 25% (Avg-1,2,3t vs HypE). Furthermore, removing deeper queries such as 2t and 3t, also results in an additional decrease by 17% (Avg-1t vs Ag-1,2,3t). The tests on different aggregation layers prove that Attention is better than average and Deepsets by 23.5% and 14.5%, respectively. Additionally, we notice that employing a trainable curvature results in a slight performance improvement of 0.3%. However, given the incremental performance boost but significant increase in the number of parameters (∼10K) that the trainable curvature adds to the framework, we ignore this component in the final HypE model.
As explained in Section 3.4, the final HypE model adopts a Poincaré ball manifold with non-trainable curvature, in addition to attention and Deepsets layer for center and limit aggregation, respectively. Additionally, HypE leverages all PFOE queries.

RQ3: Performance on Anomaly Detection
In this experiment, we utilize the entity and relation representations, trained on the DBPedia Hierarchical Taxonomy and E-commerce Product Network with query processing task, to identify products that might be anomalously categorized. We consider identifying the anomalous children by three levels of parents (i.e., taxonomy levels); 1 , 2 and 3 . The motivating application is to categorize items that are potentially mis-categorized by sellers into the more relevant (correct) part of the product taxonomy.  Table 6: Results on Miscategorized Product Anomaly Detection in E-commerce Product Networks. Best results are shown in bold and the second best results are underlined. The improvements are relative to the GQE baseline. P, R, and F1 represent Precision, Recall, and F-score, respectively. 6 We construct a pseudo-tree, where all parent nodes are infused with 10% noise of randomly sampled anomalous leaf nodes from different parts of the dataset. The goal of the model is to learn representations from this pseudo-tree and identify anomalous leaf nodes Table 7: Example of Anomalies in the E-commerce dataset. The models predict "MISCAT" and "TRUE" tags for mis-categorized and truly-categorized items, respectively. Correct and Incorrect tags are given in green and red color, respectively. HypE performs better than Query2Box (Q2B) as we consider higher level of parents because hyperbolic space is better at capturing hierarchical features. Also, HypE-SC is able to utilize semantic information to improve prediction.  The results (given in Table 5 and Table 6) show that, although HypE has comparable performance to baselines at 1 , it outperforms the baselines by more than 5% at 2 and 3 . This demonstrates the robustness of HypE to noisy data and its capability of capturing relevant hierarchical features (as F1 on 3 > 2 > 1 ) for downstream tasks. Furthermore, the specific task is critical in e-commerce search as irrelevant results impede a smooth user experience. Table  7 presents some qualitative examples from the E-commerce dataset.

RQ4: Leveraging Semantic Information
KGs generally also contain additional auxiliary information within the entities. In this section, we test the possibility of leveraging the semantic information in the DBPedia (article titles) and E-commerce (product titles) dataset to improve representations. We study two methods to connect HypE with FastText embeddings [6] of the corresponding titles: • Semantic Initiation (SI) initiates the HypE's entities with semantic embeddings and learns new HypE-SI embeddings with the query-processing task (given in Section 4.3). • Semantic Collaboration (SC) concatenates the HypE's pretrained entity representations with semantic embeddings.
We investigate the performance of these methods on the task of anomaly detection. The results of the experiments are given in Tables 5 and 6. The results demonstrate that HypE-SI shows no significant performance improvement over HypE. That is, a good semantic initialization of the vectors does not result in better representations. This is reasonable, since the semantic embeddings are learnt in the Euclidean space, and several transformations occur between the initialization and final representations. This also means that the learning framework is robust to initialization. We observe a performance improvement of 2% − 8% in case of HypE-SC when compared to HypE. This suggests the ubiquity of HypE since hierarchical representations can be independently augmented with other auxiliary features to solve more complex tasks. From the examples given in Table 7, we can observe that HypE-SI is able to leverage semantic information from product title and category name to enrich HypE's hierarchical information to produce better predictions. The additional semantic information is especially useful for product miscategorization. In the absence of semantic information, HypE will merely learn representations based on the noisy graph and will lose discriminative information between outliers and correct nodes.

RQ5: Visualization of the Poincaré ball
We extract representative examples of different operations from our dataset and visualize them in a 2-dimensional Poincaré ball of unit radius. We employ Principal Component Analysis (PCA) [5] for dimensionality reduction (R 2 → R 2 ) of the hyperboloids in Poincaré ball. Figure 9 depicts the HypE representations in a Poincaré ball manifold. Notice that the density of nodes increases superlinearly from the center towards the circumference, which is analogous to the superlinear increase in the number of nodes from root to the leaves. Thus, HypE is able to learn a better distinction between different hierarchy levels, when compared to the Euclidean distance-based baselines, which conform to a linear increase. Furthermore, we observe that hyperboloid intersections in DBPedia Taxonomy (Figure 9(a)) capture entities with common parents. Also, the disjoint nature of E-commerce Product Networks (Figure 9(b)) is illustrated by disjoint non-intersecting hyperboloids in the latent space. In addition, we can also notice that the learnable limit parameter adjusts the size of hyperboloids to accommodate its varying number of leaf nodes. Thus, the complex geometry of HypE is able to improve its precision over vector baselines that, generally, utilize static thresholds over distance of the resultant entities from query points.

CONCLUSION
In this paper, we presented Hyperboloid Embeddings (HypE) model, a novel self-supervised learning framework that utilizes dynamic query-reasoning over KGs as a proxy task to learn representations of entities and relations in a hyperbolic space. We demonstrate the efficacy of a hyperbolic query-search space against state-of-theart baselines over different datasets. Furthermore, we also show the effectiveness of hyperboloid representations in complex downstream tasks and study methods that can leverage node's auxiliary information to enrich HypE features. Additionally, we analyze the contribution of HypE's individual components through an ablation study. Finally, we present our hyperboloid representations in a 2-dimensional Poincaré ball for better comprehensibility.
( , ) (. , .) is replaced with Euclidean norm (on Query2Box representations) and hyperbolic distance ( ℎ on HypE representations) to understand the difference between the hierarchical separation of entities in the two spaces.
In the Δ results (depicted in Figure 10), we observe that, with increasing level of hierarchy the distance between entities at different levels remains constant in the case of Euclidean space and shows a clear decreasing trend for hyperbolic space. This indicates denser clustering of entities at the same level. Additionally, the Δ results, illustrated in Figure 11, depict a linear increase in distance between inter-level entities in the Euclidean space and a superlinear growth in the hyperbolic space. This shows that hyperbolic space also learns clusters such that inter-level entities are farther apart compared to Euclidean space. This nature of inter-level discrimination and intra-level aggregation demonstrates the superior ability of hyperbolic spaces at capturing hierarchical features.

B ABLATION STUDY: FINER RESULTS
The finer results, across all datasets, of our Ablation study, described in Section 4.4, are given in Tables 8 and 9. We observe that, on average over all types of reasoning queries, a combination of attention aggregation for centers and Deepsets aggregation for limits results in the best performance across all the datasets. Thus, we utilize this combination in our final model.    Table 10: Performance comparison of HypE (ours) against the baselines on an e-commerce dataset for intersection and union queries. We fix GQE (E-vector) to be our baseline and report the relative improvements against that for all the methods. The numbers are in percentages. Best results for each dataset are shown in bold.  Figure 11: Inter-level Euclidean (Q2B) and Hyperbolic (HypE) distances. Each graph presents Δ between entity pairs of a source hierarchical level, given by the graph label and the other hierarchy levels in the dataset, given on the x-axis.

C E-COMMERCE RESULTS
The intersection and union results on the e-commerce datasets, described in Section 4.3, are given in Table 10.