{'fraction_non_alphanumeric': 0.03461127322104412, 'fraction_numerical': 0.012596423309161298, 'mean_word_length': 4.685611038107753, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
HyLa: Hyperbolic Laplacian Features For Graph Learning

Due to its geometric properties, hyperbolic space can support high-fidelity embeddings of treeand graph-structured data. For graph learning, points in hyperbolic space have been used successfully as signals in deep neural networks: e.g. hyperbolic graph convolutional networks (GCN) can outperform vanilla GCN. However, existing hyperbolic networks are computationally expensive and can be numerically unstable, and cannot scale to large graphs due to these shortcomings. In this paper, we propose HyLa, a completely different approach to using hyperbolic space in graph learning: HyLa maps once from a learned hyperbolic-space embedding to Euclidean space via the eigenfunctions of the Laplacian operator in the hyperbolic space. Our method is inspired by the random Fourier feature methodology, which uses the eigenfunctions of the Laplacian in Euclidan space. We evaluate HyLa on downstream tasks including node classification and text classification, where HyLa shows significant improvements over hyperbolic GCN and other baselines.


Introduction
Graph learning is a growing area in modern machine learning. Data in many ML tasks take the form of graph structure, such as citation networks (Sen et al., 2008), social networks (Hoff et al., 2002), biological networks and etc (Rossi & Ahmed, 2015). A range of problems including node classification, link prediction, relation extraction and text classification involves handling, predicting and classification with graph data as inputs.
An important class of model proposed in graph learning is the graph convolutional neural network (GCN) (Kipf & Welling, 2016;Defferrard et al., 2016), which is a variant of convolutional networks. This class of model achieves state-of-the-art results for tasks including semi-supervised learning for node classification task, supervised learrning for graph-level classification and unsupervised learning for graph embedding. Building on the original GCN, there have been many more complex graph networks and variants developed, such as the graph attention networks (GAT) (Veličković et al., 2017), FastGCN (Chen et al., 2018), GraphSage (Hamilton et al., 2017), and others (Velickovic et al., 2019;Xu et al., 2018). In these models, nodes or features are embedded in the Euclidean space. However, embedding graphs in this way can have significant drawbacks, as it has been shown a large distortion exists for graphs embedded in Euclidean space (Chen et al., 2013;Ravasz & Barabási, 2003).
A better alternative is embedding into hyperbolic space (Cannon et al., 1997), a homogenous Riemannian manifold of constant negative curvature, which is well-suited for embedding graph-and tree-structured data due to its geometric properties. Hyperbolic space already achieves extraordinary performance on some graph tasks, such as graph embedding, link prediction and hierarchical inference (Nickel & Kiela, 2017;. It's first proposed by Ganea et al. (2018) to adopt hyperbolic geometry to develop a hyperbolic network (HNN). Later, Chami et al. (2019) and Liu et al. (2019) propose hyperbolic GCN and hyperbolic GNN separately, which take graph data as inputs. Empirical results over node classification, link prediction, molecular and chemical property prediction show the power of hyperbolic geometry for graph processing.
Real data as inputs reside in Euclidean space, and the outputs of a model (such as prediction probabilities) usually are required to be in Euclidean space as well. Therefore, the key to utilizing the power of hyperbolic geometry is the mapping from Euclidean space to hyperbolic space (pushforward mapping) and vice versa (pull-backward mapping). Unfortunately, the mappings adopted in current hyperbolic networks are numerically unstable and inefficient, with a high computation and memory budget (Yu & De Sa, 2021). Additionally, the power of hyperbolic space doesn't come at free. A issue called the NaN problem exists in most hyperbolic implementations, wherein when the space or embedding is represented with ordinary floating point numbers, as described in (Sala et al., 2018) and (Yu & De Sa, 2019), the representation error is unbounded and can grow exponentially with distance, making learning difficult. arXiv:2202.06854v1 [cs.LG] 14 Feb 2022 In this paper, we propose a different but natural approach, HyLa, to adopting hyperbolic geometry in graph learning. HyLa maps once from a learned hyperbolic embedding to Euclidean space via the eigenfunctions of the Laplace-Beltrami operator in the hyperbolic space. Our approach is inspired by the random Fourier feature method (Rahimi et al., 2007), which uses the eigenfunctions of the Laplace operator in the Euclidean space to derive features.
Our paper is structured as follows. In Section 3, we give some background on hyperbolic space and Laplace operators, with which we introduce HyLa in Section 4. We show how to use HyLa for end-to-end graph learning in Section 5. We empirically evaluate HyLa on tranductive & inductive node classification and text classification task in Section 6, where HyLa follows by a linear model outperforms state-of-the-art GCN models, with a up to 12.5% accuracy improvement in node classification task (Disease data) and 7.2% accuracy improvement in the inductive text classification task (Ohsumed data). For the first time, hyperbolic space can be used in a super easy way on graph tasks without worrying about timing and severe numerical issues. We make following contributions: • We propose HyLa, an approach to graph learning via using a hyperbolic embedding and Laplacian eigenfunctions as features. • We show how to use HyLa for end-to-end graph learning in both transductive and inductive settings. HyLa scales to large graphs and empirically outperforms previously proposed (hyperbolic) GCN models. • We show that HyLa enjoys a faster computation and less memory budget than hyperbolic GCN.

Related Work and Background on GCNs
In the graph learning scenario, we formally define a graph as G = (V, A), where V represents the vertex set consists of n nodes and A ∈ R n×n represents the symmetric adjacency matrix indicating the existence of an edge with 1 and the absence with 0. Besides the graph structure, each node in the graph has a corresponding d-dimensional feature vector: we let X ∈ R n×d denote the entire feature matrix for the whole graph. A fraction of nodes are associated with a label indicating one (or multiple) categories it belongs to. The task is to predict the label for nodes without labels or even for nodes newly added to the graph. Usually both the adjacency matrix A and feature matrix X are sparse.
A GCN layer takes A and previous-layer node representations H as inputs, and outputs f (A, H) as new node representations to the next layer. f consists of three stages: feature propagation, linear transformation and a nonlinear activation. In the feature propagation stage, the features of each node are averaged with the features of connected nodes. We can formalize this process using the "normalized" adjacency matrix S with added self-loops: where A = A + I and D is the corresponding degree matrix. The feature propagation for all nodes can then be described as SH, which encourages nodes in the same category to have similar features by propagating through connected edges. In the following stages, a GCN layer acts as a standard linear layer W with a non-linear activation function such as ReLU. Hence, we can write A deep GCN consists of a stack of multiple GCN layers. For classification tasks, similar to a MLP model, the last layer of GCN outputs the probabilities of nodes belonging to a category with a softmax classifier. However, it is observed in many tasks that a 2-layer GCN is sufficient and more layers will not help further improve the performance (oversmoothing problem (Chen et al., 2020;Zhao & Akoglu, 2019)). What's more, GCN can only be used for transductive tasks, where the test nodes are seen in the adjacency matrix during training, as opposed to the inductive setting, where no information of test nodes is used during training.
More complex variants of graph convolutional networks have been built based upon GCN. For example, GAT (Veličković et al., 2017) proposes to incorporate attention mechanism into graph networks with a masked selfattentional layer, where the averaged coefficients in the feature propagation can be controlled via trainable weights. GAT assigns different importances to nodes in the same neighborhood and helps feature propagation. Moreover, GAT is designed to be compatible in both transductive and inductive tasks. However, it requires more memory and computations than GCN. Another important variant is Fast-GCN (Chen et al., 2018), which relaxes the simultaneous availability of test data and being a model support inductive learning. Furthermore, it enables fast learning by introducing a batched training scheme and helps solve the time and memory challenges for training with large, dense graphs.
An interesting work to understand GCN is simplifying GCN (SGC) (Wu et al., 2019), where the authors remove the non-linearities in GCN and derive a simple version as: This is essentially a multi-class logistic regression on the pre-processed features S K X. Note that these features can be computed before training. This simplification does not even seem to hurt the performance of GCN, while it enables large graph learning and greatly saves memory and computation cost. Empirical results of SGC match and rival the performance of a GCN model in node classification and a series of downstream tasks.
On the other side, hyperbolic space has recently drawn more attention in graph learning and embedding. It contains more space than the Euclidean space in the sense that, the volume of a ball in hyperbolic space increases exponentially over the radius, a property similar to that of a tree, where the number of nodes increases exponentially over the depth (with a fixed branch factor). Hence, hyperbolic space is usually intuitively treated as a continuous version of a tree (Bowditch, 2006). Further, Sala et al. (2018) shows that a simple 2-dimensional hyperbolic space can embed a tree with arbitrarily low distortion. Nickel & Kiela (2017; adopt hyperbolic space for learning hierarchical representations, where it outperforms Euclidean embeddings significantly on data with latent hierarchies, in terms of both representation capacity and generalization ability.
The push-forward and pull-back mappings are the core of utilizing the power of hyperbolic geometry. This is needed not only because the features are sometimes required to be Euclidean, but also because many commonly used operations in the Euclidean space-including multiplication, addition and convolution-are not well defined in the hyperbolic space. Therefore, when such an operation is needed, one has to move from the hyperbolic space to the Euclidean space with a push-forward mapping, apply the operations in the Euclidean space, then map it back to the hyperbolic space with the pull-backward mapping so as to maintain the hyperbolic geometry and hyperbolic representations.
However, there is no invertible isomorphism between both spaces as the hyperbolic space is a completely different space to the Euclidean space. Rather, note that the tangent space of the hyperbolic space is Euclidean, which is a real vector space that intuitively contains the possible directions in which one can tangentially pass through a point in the hyperbolic space, i.e. where gradients lie. Fortunately, there are maps between the hyperbolic space and its tangent space, namely the exponential map exp z (·) from the tangent space to the hyperbolic space and the log map log z (·) from the hyperbolic space to the tangent space. As a result, the exponential and log maps are adopted as the push-forward and pull-backward mappings between Euclidean and hyperbolic space in all the current hyperbolic networks (Liu et al., 2019;Ganea et al., 2018;Chami et al., 2019).
The shortcomings of using them falls into three aspects: (1) exp z (·) and log z (·) vary and take different forms as z moves in the hyperbolic space, as a result exp o (·) and log o (·) centered at the origin o are usually chosen to be the push-forward and pull-backward mappings for simplicity and efficient implementations; (2) even with exp o (·) and log o (·), the function is still complicated and consists of hyperbolic functions such as sinh, cosh, which are numeri-cally unstable and ill-conditioned; and (3) the push-forward and pull-backward mappings are used at every hyperbolic layer, which incurs a high computation complexity in both the model forward and backward loop. Hence, existing hyperbolic networks cannot scale to large graphs.
Another line of work focuses on the numerical issues of using hyperbolic geometry: namely, the "NaN problem" (Yu & De Sa, 2019). When the hyperbolic space/embedding is represented or stored with ordinary floating-point numbers, the representation error will increase unbounded as the embedding gets further away from the origin. In practice, when adopting hyperbolic geometry such as running hyperbolic networks, because of the hyperbolic functions and the NaN problem, one needs to carefully deal with different clippings so as to avoid NaNs and Infs. This potentially prevents hyperbolic embeddings from being far away from the origin and limits the power of hyperbolic space. Although some theoretical and practical solutions have been provided (Yu & De Sa, 2021;Sarkar, 2011), they are far from adoption in existing hyperbolic networks, which further prevents hyperbolic geometry from widely used.
In this work, we use the eigenfunctions of the Laplace-Beltrami operator in hyperbolic space to derive features. The Laplacian is an important object to study functions defined in a space. In the Euclidean space, the Fourier basis functions are eigenfunctions of Euclidean Laplacian, which enjoys a great usage in many area (Rahimi et al., 2007). In the discrete case, the graph Laplacian is the key of deriving GCN, where the feature propagation is interpreted as applying a first order filter over the spectrum of graph Laplacian (Kipf & Welling, 2016;Defferrard et al., 2016). Through analysis of the spectrum of graph Laplacian, different variants of the filter S are proposed for feature propagation.

Hyperbolic Space and the Laplacian
Hyperbolic space. n-dimensional hyperbolic space H n is studied and used with a model: a representation of H n within a Euclidean space. Commonly used models include the Poincare ball model and the Lorentz hyperboloid model (Yu & De Sa, 2019;Nickel & Kiela, 2017). For example, (Ganea et al., 2018) derives HNN using the Poincare ball model, while (Chami et al., 2019) uses the Lorentz hyperboloid model to develop HGCN. We develop our approach in this paper using the Poincare ball model, but our methodology is independent of the underlying model and can be applied to other models.
The Poincaré ball model of H n is the Riemannian manifold (B n , g p ) with B n = {x ∈ R n : x < 1} being the open unit ball and the Riemannian metric being g p (x) = 4(1 − x 2 ) −2 g e , where g e is the Euclidean metric. The corresponding metric distance on B n is The Poincaré ball model has a convenient parameterization and can serve as a clear visualization of the hyperbolic space due to its conformality: angles in the Euclidean space are the same as they are in the actual hyperbolic space.
Laplace operator. The Laplace operator ∆ is defined as the divergence of the gradient of a scalar function on Euclidean space. It serves as an important characterization of the space, which has a physical interpretation for nonequilibrium diffusion as the extent to which a point represents a source or sink of chemical concentration.
The Laplacian is self-adjoint, and its eigenfunctions are the solutions of the Helmholtz equation If the domain Ω of this equation is bounded in R n , then the eigenfunctions of the Laplacian form an orthonormal basis for the Hilbert space L 2 (Ω) (Gilbarg & Trudinger, 2015).
In Euclidean space, an important set of eigenfunctions of the Laplacian is the Fourier basis functions x → sin(u T x) and x → cos(u T x), which are used throughout the machine learning literature. For example the random Fourier features proposed in (Rahimi et al., 2007) use these basis functions for randomized, low-dimensional approximations of kernel functions, which are then used in many other applications including DNN nonlinearities (Sitzmann et al., 2020).
Laplace-Beltrami operator. The Laplace-Beltrami operator L (also sometimes denoted ∆) is the generalization of the Laplace operator to Riemannian manifolds. It is defined as the divergence of the gradient for any twice-differentiable real-valued function f on a Riemannian manifold.
We are interested in the Laplace-Beltrami operator in the hyperbolic space (Agmon, 1987). In the 2-dimensional Poincare disk model D = B 2 , the Laplace-Beltrami operator takes the form More generally, in n dimensions, this operator has the form The real eigenfunctions of the Laplace-Beltrami operator take on the role of the Fourier basis in Euclidean space. In particular, any sufficiently nice function can be written as an (infinite) linear combination of these eigenfunctions. Another nice property of these eigenfunctions is that they are invariant to isometries of the space: any isometric transformation of an eigenfunction e µ,b yields another eigenfunction with the same µ but a transformed b (depending on how the isometry acts on the boundary ∂D).

HyLa: Features from Eigenfunctions
Motivated by their utility as a basis for "nice" functions in hyperbolic space, we propose to use the eigenfunctions as features for points embedded in H n . This idea is complicated somewhat by the fact that the choice of eigenfunctions is not unique: there is an entire subspace of infinitely many solutions to the Helmholtz equation for any given eigenvalue and this subspace can be parameterized in many different ways. The way we parameterize this matters if, as we will do in Section 5, we are going to backprop through the eigenfunctions. In this section, we introduce HyLa features, a parameterization of the real eigenfunctions of hyperbolic space in the Poincaré ball model.
We start our derivation by introducing the function where ∂B n denotes the boundary of the ball. This function is equal to the Poisson kernel in two dimensions (but not for n = 2). It is easy to verify (Agmon, 1987) that for µ ∈ C, if e µ,b (z) = P (z, b) µ then Le µ,b = µ(µ − n + 1)e µ,b .
Just as in the Euclidean case, where we are more interested in bounded eigenfunctions of ∆ like x → cos(u T x) than we are in eigenfunctions like x → exp(u T x), here we want to restrict our attention to cases where µ(µ − n + 1) is negative. To achieve this, we consider µ = n−1 2 + i λ 2 , for real λ, which yields the real negative eigenvalue of L µ(µ − n + 1) = − 1 4 (n − 1) 2 + λ 2 .
Unfortunately, these µ are complex, and as are the corresponding e µ,b . We want real eigenfunctions. Fortunately, we can easily get real eigenfunctions from complex ones by observing that the Helmholtz equation for real λ is preserved under complex conjugation, and as a consequence the real part of any of these eigenfunctions e µ,b (which is just the average of it and its conjugate) will also be an eigenfunction with the same eigenvalue-this will also hold for the real part of e µ,b scaled by any complex phase exp(ic). We write this explicitly for z ∈ B n , b ∈ ∂B n , λ ∈ R, and c ∈ R as This parameterization, which we call HyLa (for HYperbolic LAplacian features), is the one we will use to produce features for graph learning. HyLa eigenfunctons have the nice property that they are bounded in almost every direction, as P (z, b) approaches 0 as z approaches any point on the boundary of B n except b.

End-to-End graph learning with HyLa
In this section, we give a full recipe for using HyLa for graph learning. In a graph learning task, we assume the adjacent matrix A is given, sometimes together with a node representation feature matrix X. HyLa can be used for end-to-end graph learning at both node and feature level.
The high-level idea of HyLa is simple. For each object (either nodes or features) we initialize an embedding z of that object in hyperbolic space at a point near the origin, by sampling each coordinate z i at random from [−10 −5 , 10 −5 ].
We randomly sample some constants for HyLa features by sampling the boundary points b uniformly from the boundary of the ball, eigenvalue constants λ from a zero-mean s-standard-deviation Gaussian and biases c uniformly from [0, 2π]. These constants remain fixed throughout training, while the embedding points z are allowed to be learned. The full setup is described in Algorithm 1.
Node level. When used at node level, we initialize a low dimensional hyperbolic embedding Z for the whole graph and then derive a high dimensioanl HyLa H as the node feature matrix instead (this can be stacked with X if it exists). This method is better-suited for tasks where no feature matrix is available or meaningful features are hard to derive. However, the size of Z will be proportional to the size of the graph, hence it cannot scale to very large graphs due to memory and computation constraints. Further more, this method can only be used in transductive setting.
Feature level. When used at feature level, we assume a feature matrix X is provided, and we initialize a low dimensional hyperbolic embedding z for each feature (not each node) and derive a high dimensional feature level HyLaĤ, using H = XĤ as the node feature matrix instead. Typically, the node feature dimension is fixed and much lower than the number of nodes in a graph, so this method can be used even for large graphs. What's more, since the hyperbolic embeddings are built for each node feature dimension, it can be used in both transductive and inductive setting, as long as the data shares the same lower level features to those of the training data. However, it's conditioned on the existence of a feature matrix and that the feature matrix takes sufficient information for learning the task, otherwise, there would be performance degradation.
End-to-end learning. In either the node-level or featurelevel case, an end-to-end model can be formalized as f (A, H). In our experiment, we use HyLa together with SGC, which takes the form Y = softmax(A K HW), for a trainable weight matrix W. This is essentially a linear model with respect to W, i.e. logistic regression. Furthermore, A K or A K X can be pre-computed in the same way as SGC does before training and inference.
Training details. We use cross-entropy as the loss function and joint optimize the low dimensional hyperbolic embedding E and linear weight W simultaneously during training. Specifically, we use the Riemannian SGD optimizer (Bonnabel, 2013) for E and Adam (Kingma & Ba, 2014) optimizer for W. RSGD naturally scales to very large graph because the graph connectivity pattern is sufficiently sparse. We adopt early-stopping for Adam optimizer as regularization. We tune the hyper-parameter via grid search over the parameter space.
We provide Algorithm 1 as a way to deploying HyLa to an end-to-end graph learning system. Worthy to note, HyLa shares some similarities to neural networks with first layer using sin, cos as activation functions, e.g. a linear embedding follows by a cos nonlinearity (Luo et al., 2018;Sibi et al., 2013).

Experiments
We first evaluate HyLa on node classification tasks on citation networks and social networks. Then we extend it to text classification tasks in both transductive and inductive settings.

Node Classification
We evaluate HyLa on the semi-supervised node classification task, where the goal is to classify each node into a correct category. We use both transductive and inductive datasets detailed below.
1. Citation Networks. Cora, Citeseer and Pubmed (Sen et al., 2008) are standard citation network benchmarks, where nodes represent papers, connected to each other via citations. We follow the standard splits (Kipf & Welling, 2016) with 20 nodes per class for training, 500 nodes for validation and 1000 nodes for test.
2. Disease propagation tree (Chami et al., 2019). This is tree networks simulating the SIR disease spreading model (Anderson & May, 1992), where the label is whether a node was infected or not and the node features indicate the susceptibility to the disease. We use dataset splits of 30/10/60% for train/val/test set.
3. Airport. We take this dataset from (Chami et al., 2019). This is a transductive dataset where nodes represent airports and edges represent the airline routes as from OpenFlights. Airport contains 3,188 nodes, each node has a 4 dimensional feature representing geographic information (longitude, latitude and altitude), and GDP of the country where the airport belongs to. For node classification, labels are chosen to be the population of the country where the airport belongs to. We use dataset splits of 524/524 nodes for val/test set.

4.
Reddit. This is a much larger graph dataset built from Reddit posts, where the label is the community, or "subreddit", that a post belongs to. Two nodes are connected if the same user comments on both. We use a dataset split of 152K/24K/55K follows (Hamilton et al., 2017;Chen et al., 2018), similarly, we evaluate HyLa inductively by following (Wu et al., 2019): we train on a subgraph comprising only training nodes and test with the original graph.
For Reddit dataset, we compare to the reported performance of GaAN, supervised and unsupervised variants of Graph-SAGE (Hamilton et al., 2017), SGC (Wu et al., 2019) and FastGCN (Chen et al., 2018) in Table 3. Note that GCN and HGCN can not be trained on the Reddit dataset because the whole adjacency matrix is too large to fill into memory. Worthy to mention, the HyLa-SGC model can also be designed to train in a batch by batch way for datasets that are even larger than Reddit in case of OOM.
Results. From our results in Table 3, we conclude that HyLa is particularly strong and expressive for graph learning. Together with the linear SGC, HyLa-SGC outperforms state-of-the-art GCN models on most datasets. On the Cora dataset, HyLa-SGC beats vanilla SGC and GCN/HGCN and can match the performance of GAT. Particularly for the tree network disease with lowest hyperbolicity (more hyperbolic), the improvements of HyLa-SGC to SOTA HGCN is about 12.5%! The results suggest that the more hyperbolic the graph is, the more improvements will be gained with HyLa.
On Reddit dataset, Table 3 shows that HyLa-SGC outperforms the sampling-based GCN variants, SAGE-GCN and FastGCN by more than 1%. However, the performance is close to SGC, which may indicate that the extra weights and Table 3. Test accuracy/Micro F1 Score (%) averaged over 10 runs on node classification task. Performance of some baselines are taken from their original papers. OOM: Out of memory. nonlinearities are unnecessary for this particular dataset.

Dataset
Visualization In Figure 1, we visualize the learnt node HyLa features on Cora, Airport and Disease datasets with t-SNE (Van der Maaten & Hinton, 2008) and PCA projection, where HyLa achieves great label class separation (indicated by different colors).

Transductive Text Classification
We further evaluate HyLa on the transductive text classification task to assign labels to documents. We conducted experiments on 4 standard benchmarks including R52 and R8 of Reuters 21578 dataset, Ohsumed and Movie Review (MR) follows the same data split as (Yao et al., 2019;Wu et al., 2019). A detailed dataset statistics is provided in Table 2.
Experiment Setup Previous work (Yao et al., 2019) and Wu et al. (2019) apply GCN and SGC by creating a corpuslevel graph where both documents and words are treated as nodes at the same level in the graph. For the weights and connections in the graph, word-word edge weights are calculated as pointwise mutual information (PMI) and worddocument edge weights as normalized TF-IDF scores. The weights of document-document edges are unknown and left as 0. We follows the same data processing setup for this task, and adopt HyLa at node level since only adjacent matrix is available. We train the HyLa-SGC model for a maximum of 200 epochs and compare it against TextSGC and TextGCN in Table 4.
Performance Analysis. When HyLa is adopted at node level, it can match the performance of GCN and SGC. The corpus-level graph may contain sufficient information to learn the task, and hence HyLa-SGC does not seem to outperform them, but still has a comparable performance.

Inductive Text Classification
In practice, the inductive setting is more common where the test set may not be available at training time, or the model is tested on totally new nodes. Standard GCN and SGC structure cannot solve this particular problem as they require the knowledge of full adjacency matrix during training time including both training and test nodes. What's more, the approach adopted in the transductive setting where a corpus graph is built with both documents and words at the same level, will cost more memory and also being harder to embed as documents and words have different semantic meanings, which might be detrimental to the performance. TextGCN can be made inductive by replacing GCN with FastGCN in the model as proposed in (Yao et al., 2019).
We show that HyLa can be easily used in an inductive setting for text classification. We take the sub-matrix of the large matrix in the transductive setting, including only the document-word edges as the node representation feature matrix X, then follow the procedure in Section 5 to adopt HyLa at the feature level to get H.

Experiment Setup
Since the weight of documentdocument edges are unknown, i.e. the adjacency matrix of documents is unknown, we use a logistic regression layer (LR) follows HyLa directly as Y = softmax(XHW) to be our model. We compare our HyLa-LR model with inductive TextGCN and a simple logistic regression model, i.e. Y = softmax(XW) in Table 5.   Table 5 shows the extraordinary performance of HyLa for inductive text classificaion task. When HyLa is used at feature level, together with a linear regression model it can already outperform inductive TextGCN, sometimes even better than the performance of a transductive TextGCN model, which indicates that there is redundant information in the corpus-level graph.

Performance Analysis
From the results on the inductive text classification task, we argue that HyLa (at feature level) is particularly useful in the following three ways. First, it can solve the OOM problem of classic GCN model in large graphs, and requires less memory during training, since there are limited number of lower level features (not to say that X is usually sparse), and there is no need to build a corpus-level graph anymore. Second, it is naturally inductive as HyLa is built at feature level (for each word in this task), it generalizes to any unseen new nodes (documents) that uses the same set of features (words). Third, the model is simple: HyLa follows by a linear regression model, which computes faster and easier than classical GCN models.

Efficiency
Following Wu et al. (2019), we measure the training time of HyLa-based models on the Pubmed dataset, and we compare against state-of-the-arts graph networks including SGC, GCN, GAT, HGCN and HNN. In Figure 2, we plot the timing performance of various models. In particular, we take into account the pre-computation time of some models into training time. We measure the training time on a NVIDIA GeForce RTX 2080 Ti GPU and show the specific timing statistics in Appendix.
Particularly for the HGCN model, in order to achieve the report performances, we follow the same training procedure using their public code, which is divided into two stages: (1) conduct a link prediction task on the dataset to learn hyperbolic node embeddings, and (2) use the pretrained embeddings in the last stage, then train a MLP classifier. Hence, we add the timing of both stages as the timing for HGCN. Figure 3 shows that HyLa-based models achieves the best performance while incurring a minor computational slowdown, which is 4.4× faster than HGCN.

Conclusion
We theoretically derive HyLa as an expressive feature for graph learning using the eigenfunctions of Laplacian in the hyperbolic space. Empirical results of adopting HyLa on node classification and text classification tasks show that HyLa-based models can achieve SOTA performances in a simple and efficient way. HyLa sheds light on utilizing hyperbolic geometry into neural networks for graph processing in an entirely different approach to previous work.
Some future directions include (1) using HyLa together with non-linear graph networks such as GCN to derive more expressive models; (2) adopting numerically stable representations of the hyperbolic space to avoid potential NaN problem of running HyLa-based models.  For this task, we adopt HyLa at feature level for most datasets except from the Airport, whose features are only 4dimensional. We use the model softmax(A K HW), where HyLa matrix H ∈ R n×d1 is derived from the hyperbolic embedding Z ∈ R n×d0 and randomly sampled d 1 boundary points over the unit sphere. Another important hyper-parameters in Algorithm 1 is s: eigenvalue constants λ are sampled from a zero-mean s-standard-deviation Gaussian. During training, we adopt Riemannian SGD (RSGD) optimizer of learning rate lr 1 for Z and Adam optimizer of learning rate lr 2 for W. We joint optimize both Z, H together. We tune all the hyper-parameters via grid search and provide it in Table 6.

A.2. Text Classification
For this task, we set K = 2 and train the model for a maximum of 200 epochs without using any regularization (e.g. early stopping). Other hyper-parameters in both transductive and inductive setting are shown in Table 7. Particularly note that in the transductive setting, HyLa is used at node level, hence the size of parameters will be proportional to the size of graph, in which case, d 0 and d 1 can not be too large so as to avoid OOM. In the inductive setting, there is no such constraint as the dimension of lower level features is not very large itself. We will open source our code for reproducibility.

B. Timing
We show the specific training timing statistics of different models on Pubmed dataset in Table 8.