{'fraction_non_alphanumeric': 0.03565466253279275, 'fraction_numerical': 0.008854042451705223, 'mean_word_length': 4.8400069637883005, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
R-GCN: The R Could Stand for Random

The inception of the Relational Graph Convolutional Network (R-GCN) marked a milestone in the Semantic Web domain as a widely cited method that generalises end-to-end hierarchical representation learning to Knowledge Graphs (KGs). R-GCNs generate representations for nodes of interest by repeatedly aggregating parameterised, relation-specific transformations of their neighbours. However, in this paper, we argue that the the R-GCN's main contribution lies in this"message passing"paradigm, rather than the learned weights. To this end, we introduce the"Random Relational Graph Convolutional Network"(RR-GCN), which leaves all parameters untrained and thus constructs node embeddings by aggregating randomly transformed random representations from neighbours, i.e., with no learned parameters. We empirically show that RR-GCNs can compete with fully trained R-GCNs in both node classification and link prediction settings.


Introduction
Knowledge Graphs (KGs) are the ideal data structure to represent both expert knowledge and observational data, and can be used to reveal new insights about the modelled domain. The latter becomes even more true thanks to today's hybrid Machine Learning (ML) methodologies [1,2] where semantically enriched data is combined with a statistical method's ability to learn. As these models typically operate on Euclidean data, the integration of KGs into their decision making processes involves a non-trivial transformation from information represented as a variable number of nodes and edges to fixed size numerical vectors, i.e., graph embeddings. Relational Graph Convolutional Networks (r-gcns) embed semantic information about an entity contained in a KG by iteratively updating node representations. First, the previous representations for a node's neighbours are collected and passed through a learned, relation-specific, transformation. These transformed representations are referred to as "messages". Second, an aggregation of the collected messages and the node's own state yields the updated node representation [3].
In this work, we ablate the parameterisation of the r-gcn's message passing step by randomly initialising and freezing the relation-specific transformations. We evaluate these "Random Relational Graph Convolutional Networks" (rr-gcns) on nine node classification datasets, and empirically show that these random transformations produce embeddings for node classification that are on par with -and sometimes even outperform -those produced by r-gcns. We also compare the embeddings produced by our rr-gcns to an end-to-end trained rgcn for a link prediction task and obtain very similar results.
The remainder of the paper is structured as follows. In Section 2, we summarise our main contributions to the field of KG embedding. Section 3 provides the necessary background in concepts that are fundamental to our rr-gcn. Next, in Section 4, we discuss our technique formally. We then compare rr-gcn to the original r-gcn and discuss the implications of these results in Section 5. Finally, we provide some closing remarks in Section 7 and discuss future work in 8.

Contributions
This paper is the first to evaluate fully random r-gcns for KG embeddings. We show that random transformations can capture a surprising amount of information. On the one hand, this makes our rr-gcns an interesting baseline model when developing trained embedding methods. On the other hand, the effectiveness of these random transformations illustrates that, for KGs, the rgcn's message passing and aggregation paradigm is more significant than the actual parameters, which have to be obtained through an expensive training procedure. This opens up avenues for further research on more efficient, and more powerful, message passing parameretisations for KGs.

Background
In this section, we touch upon the concepts that are fundamental to our rr-gcn approach: (i) we briefly outline prior research directions in Knowledge Graph Embedding (KGE); (ii) we provide the necessary theory of r-gcns; and (iii) we discuss the concept of learning from random transformations.

Representation Learning for KGs
Techniques to embed substructures in KGs can be categorised into four groups. A first category compromises of techniques that extract generic properties from neighbourhoods of substructures of interest. These can either be feature-based [4,5] or exploit similarities with other substructures of interest, i.e., kernel functions [6,7]. A second category consists of (algebraic) embedding spaces learned using tensor factorisation or through negative sampling [8,9,10,11,12]. A third category adapts existing natural language processing (NLP) techniques, such as Word2Vec [13], to graph structures [14]. A fourth and final category contains the message passing architectures that are trained end-to-end to aggregate relevant information around the substructures of interest [15,16,17,18]. r-gcns belong to this final category.

R-GCN
Graph Neural Networks (gnns) allow for graph ML by learning how to update a node's representation based on its neighbours. These gnns work analogously to classical neural networks: they start from initial node features which are passed through multiple gnn layers to refine and abstract the representations by mixing in information from one additional hop with every layer.
A single gnn layer operates in three steps: every node (1) generates a "message" based on its current representation and sends it along its outgoing edges, (2) aggregates incoming messages, and (3) updates its representation based on the aggregated messages and its own previous representation.
The definitions of the message, aggregation and update functions differentiate several subtypes of gnns. The most prominent of these subtypes, the Graph Convolutional Network (gcn) [19], uses a learned transformation matrix as its update function, aggregates by averaging messages (including the node's own message) and passes the aggregations through an activation function to yield new node representations (see Equation 1).
Here, φ denotes the activation function used, h (l) i is the representation for node i at layer l, N (i) is the set of neighbours for node i, and W (l) is the learned transformation matrix in layer l. This formulation assumes unweighted edges.
r-gcns extend gcns to support typed edges. As KGs are multi-relational, this extension is particularly interesting for hybrid ML. They are almost identical to standard gcns but learn a different transformation matrix W (l) r for every relation r ∈ R. A separate transformation W (l) 0 takes care of self-loops (see Equation 2). Note that R contains two copies of every relation in the original KG, as r-gcns also take outgoing (i.e. inverse) edges into account, with different learned transformations.
Learning separate transformations per relation results in extra parameters and might not work well for sparse types. Weight sharing between relations has been proposed to reduce overfitting and the number of required parameters [15]. Using a basis decomposition, a layer's parameters are reduced to a fixed set of base parameter matrices V i (Equation 3).

Learning from Random Representations
Our approach is inspired by an algorithm from the time series classification domain, rocket [20,21]. This algorithm applies a large number of random convolutions and aggregations to the raw time series input to extract features with state-of-the-art predictive performance. These random transformations do not have to be learned and can be applied very quickly, making rocket more scalable than other time series classification techniques. Two types of aggregations are used within rocket: (i) max-pooling, which just retrieves the maximum value of a certain convolutional filter when slided across the time series, and (ii) the proportion of positive values (ppv), which, for every filter, captures the proportion of the input time series for which the output of the convolution operation is positive. We will discuss the latter aggregation method in more depth in the context of graphs in Section 4.2.
The idea of freezing, or not training, layers in neural networks is not novel either. Reservoir Computing [22] is a paradigm in recurrent neural networks where inputs are first passed through any black-box non-linear system called a "reservoir", which could be an untrained neural network. As such, only the final layer that maps the learned representations to the target output is trained. The paradigm of random modelling often introduces a trade-off between efficiency and effectiveness. As many random, albeit less effective, transformations can often be applied very efficiently, they allow to scale to higher numbers. An example of a technique where quantity matters over quality is ExtraTrees [23], where a large number of decision trees with random splits yield good results.
In a blog post, detailing the inner workings of (non-relational) gcns [24], Thomas Kipf hinted at the discriminative power of representations resulting from untrained transformations, but this hypothesis was never formally evaluated, let alone for KGs. In a recent paper [25], random transformations were evaluated in the context of link prediction in a neuromorphic computing setting. In this research, the authors backpropagated through random weights to train initial node embeddings. Other research demonstrated that keeping the initial node features random and frozen, and only training the message passing transformations results in good performance for unirelational graphs [26,27]. For KGs, it is even possible to deterministically generate meaningful node features, avoiding the need to learn initial embeddings in the first place [28]. As will be explained in the remainder of this paper, we are the first to explore fully random networks for KGs, that keep both the initial representations and the transformations frozen.

Methodology
In this section, we elaborate on the modifications we made to the original r-gcn algorithm. We also present a new information aggregation function for graphs: ppv. Finally, we compare memory requirements for trained and random r-gcns.

Modifying the R-GCN Layer
Our rr-gcn layers use the same message paradigm as r-gcns (see Equation 2). The most important difference is that we randomly initialise all transformation matrices using the Glorot uniform initialisation function [29] and then keep them fixed to freeze the network. Basis decomposition is no longer required and thus not used, as overfitting cannot occur since no training is involved.
In [15], the node features that are fed to the first r-gcn layer are one-hot encoded. Using one-hot node features effectively assigns a separate initial embedding matrix W (0) r per relation type in the first message passing layer. This is indeed desirable for r-gcns, since the output node representation dimensionality is usually very small (10 to 16 in the original paper [15]). Starting with small, randomly initialised, node representations limits the level of detail in the initial node characterisations and leads to worse performance for some datasets [30]. As our rr-gcns use much larger node representations, we can feed our first message passing layer random features h (0) i ∈ R d , with d the embedding size, -which we also do not train -and save them as a single seed. In our experiments, this did not impact performance compared to one-hot encoded inputs.

Proportion of Positive Values (PPV)
The performance of rocket's random feature extraction for time series depends strongly on their use of ppv [20] pooling, as is also apparent from their second version which abandons max-pooling and uses only ppv features [21]. Inspired by their success, we adapted a variant of ppv to graphs that aims to encapsulate additional information about a node's neighbourhood. Given a matrix of node representations H, we define ppv as the proportion of strictly positive values in a 1-hop neighborhood per representation dimension (see Equation 4).
Here, h i is the representation for the i th node (i th row in H) and 1[·] is the indicator function. The resulting matrix P , with the same dimensionality as H, houses additional node features that capture information about the feature diversity in every node's neighborhood. The ppv function does not differentiate neighbours based on relation types as it is applied as a "post-processing" step to representations that are the result of relation-specific transformations.

Putting it all Together
Given a KG G = (V, E, R) with entities v i ∈ V, edges (v i , r, v j ) ∈ E and relation types r ∈ R, we construct Euclidean vectors for every entity. These vectors should encapsulate each entity's semantics, as contained in the triples of the KG, such that they can be used as features for machine learning models. Note Algorithm 1: Generating node embeddings using rr-gcn layers Inputs: relations R, per-relation node neighbours N , embedding size e, number of layers n, seed s, nodes V Output: Matrix ∈ R |V|×e 1 Function embed-rrgcn-ppv(e, s, n): that entities can also be "literals" and have associated values. Both r-gcns and the proposed rr-gcns, however, regard literals as ordinary nodes. Algorithm 1 illustrates the process of generating these embeddings in pseudocode. For ease of notation we assume that triples are accessible through N r (i), which contains the neighbours connected to an entity v i through a relation r ∈ R. We also assume that the relations in R are encoded as integers starting at one. Apart from the KG's characterisation, our method takes an embedding size e, a seed s, and a number of layers n as input. Based on the seed s, random initial embeddings are sampled from a normal distribution with variance chosen such that the sum of a node's embedding is a standard normal random variable. We chose this initialisation strategy over Glorot as the range of Glorot-initialised matrices gets smaller with both the number of rows and columns. This makes sense for transformation matrices, but not for embeddings. The seed s is then reused to generate a list of additional random seeds, one for every transformation matrix W i . This list of seeds is given as an argument to the rrgcn-conv function, listed in Algorithm 2.

Algorithm 2: rr-gcn Message Passing
Inputs: list of seeds S, embedding size e, node representations H Output: Matrix ∈ R |V|×e 1 Function rrgcn-conv(S, e, H): The rrgcn-conv function implements the r-gcn's message passing equation (see Equation 2) with optimised memory usage. Relation-specific (and selfloop) contributions are iteratively added to the zero-initialised (O e×e ) output representation matrix X, generating the required transformations on-the-fly from the given seeds. Note that x i and h i stand for the i th row of X and H respectively, and that the computations for these rows are done all at once using efficient matrix multiplications; we only use the row-wise notation for clarity.
After the first message passing round, the obtained representations are passed to a ReLU non-linearity ( + ) and stored in H. These hidden representations are then used to calculate the one-hop ppv features P as described in Section 4.2. The ReLU activation function is not used for ppv representations as they are positive by definition and the result of a non-linear operation.
Using the updated hidden node features, the previous steps are repeated n − 1 times. There are two noteworthy details here: (i) we use the same seeds for every convolution, and (ii) we convolute the ppv features independently from the regular representations (but with the same seeds). In our experiments, we noticed that using different transforms in every layer was not necessary -and sometimes even hurt performance -for rr-gcns.
After the random message passing layers, the concatenation ( ) of H and P along the horizontal axis yields the final node embeddings.

Memory Usage
Trained r-gcns need to store their parameters in (GPU) memory. Since relationspecific transformations are typically quite small, this is dominated by the initial node representations. Especially when using one-hot encodings as initial representations, this memory load can become quite significant. Additionally, in the backward pass, gradients for all weights (and for optimisers such as Adam [31], even gradient moments) have to be stored as well, which at least doubles the parameter memory requirements. For a KG with nodes V, an r-gcn with B bases and an initial representation size e thus needs to be able to keep at least two B × |V| × e-dimensional float-tensors (4 bytes per element) in memory. For DBLP (a dataset with 4,470,778 nodes, Table 2) and a 40-base r-gcn with a 16-dimensional embedding size this results in at least 22.89GB of memory.
An even bigger source of memory usage for trained r-gcns, is the storage of intermediate activations. Even if per-relation contributions are iteratively summed to an accumulator as in Algorithm 2, by default, all individual contributions accross all layers are still kept in memory during training, as they are required for backpropagation. This results in a |V| × e l -dimensional float-tensor for every relation r ∈ R and the layer's global output, for every layer l, with e l the output dimensionality for layer l. For DBLP (136 relations with inverses) and a single r-gcn layer with a 16-dimensional output, e.g., this results in 39.20GB. Activation checkpointing [32] could be used to avoid storing intermediate activations by recalculating them during the backward pass, trading in memory for compute, but no current implementations use this approach.
Since rr-gcns do not need a backward pass, we do not need to keep any gradients or intermediate activations in memory. Moreover, because the initial node embeddings and transformation matrices are random, we also do not keep them in memory; storing the seeds is sufficient to recreate the necessary tensors when they are needed. The peak memory usage for an rr-gcn with embedding size e is dominated by (1) the previous-layer node representations, (2) the previous-layer ppv features, (3) the accumulator matrix X (Algorithm 2) and (4) the intermediate results for a single relation type; four |V| × e-dimensional float-matrices, which is comparable to the memory complexity of an r-gcn's forward pass during inference. rr-gcns clearly need much less memory than their trained counterparts for a given embedding size. However, as these embeddings result from combinations of randomly transformed random representations, to capture useful information, a larger embedding size is needed compared to rgcns, which are trained to select only the useful features for the downstream task. As such, rr-gcns are not necessarily more memory efficient than r-gcns, depending on how many random features are necessary for the downstream task. For DBLP and an r-gcn with embedding size 512, e.g., the resulting peak memory usage is 36.62GB. It is important to note that, once trained, r-gcn inference actually requires less memory because of the more compact representations.

Results
In this section, we present empirical results that show how our rr-gcn method matches -and in some cases exceeds -the performance of end-to-end trained networks for the two main KG machine learning tasks: node classification and link prediction, which we first briefly introduce along with the datasets used for evaluation. We conclude the section with a visual exploration of the resulting embedding spaces.

Node Classification
For node classification, we are provided with two subsets of vertices V tr ⊆ V and V te ⊆ V with corresponding labels y tr and y te . The goal of this task is to construct a model or hypothesis h(.) based on V tr , y tr , and the graph's edges E that minimises a loss function L(.) and generalises well to the unseen labeled vertices in V te : arg min h L(y te , h Vtr,ytr (V te )) A first collection of datasets we use to compare rr-gcns and r-gcns are those published by Ristoski et al. [33]. These four datasets have varying sizes, but most of them are of a rather small scale. We include them because they were used to evaluate node classification in the original r-gcn paper [15]. Table 1 summarises some important properties of the four datasets.
In addition, we compare both techniques on "kgbench" [34], which contains five larger scale KGs.

Link Prediction
The goal of link prediction is to infer true triples that are not yet captured in a KG. To that end, the KG's known triples are split in two subsets E tr ⊆ E and E te ⊆ E and a model or hypothesis h(.) is constructed. This hypothesis, or "scoring function" is trained to assign a high output to the true edges E tr ⊆ E and a low output for corrupted, false edges E tr,c ⊂ E. The trained hypothesis should then be able to recover the withheld edges E te by assigning them high scores, and low scores to all false edges, i.e., it should minimise a loss function L(.): We evaluate our rr-gcn's link prediction performance only on the FB15k-237 dataset, as the other datasets used for evaluation in the original r-gcn work [15] have all since been found to contain significant leakage [11]. Table 3 summarises some important properties of this dataset.

Evaluation
We implemented our rr-gcn layer and embedder in PyG [35], an extension of the popular deep learning framework PyTorch [36] that facilitates the implementation of message passing networks. PyG provides parallel execution of node representation updates. In the remainder of this section, we discuss the evaluation setup for the node classification and link prediction tasks, and introduce our methodology for a rudimentary qualitatitve analysis of the embeddings.
Node Classification A first step in our evaluation procedure is to reduce the size of the KG by excluding vertices that are further than n hops away from any of the training (V tr ) or testing (V te ) vertices, as no information from more than n hops away can be propagated to the nodes of interest with n message passing layers. Once the size of the KG is reduced, we apply n layers of our rr-gcn to create embeddings of size e. The constants n and e are tuneable hyper-parameters. Once the embeddings are generated, they are provided as input to a gradient boosting classifier. In this study, we used CatBoost [37].
For the small-scale benchmark KGs, the hyper-parameters n and e were tuned using a grid search with stratified 5-fold cross-validation, with a different rr-gcn seed in every fold. For the larger-scale KGs, we evaluated an rr-gcn five times with different seeds on the provided validation set to tune the hyperparameters. We tuned n to take a value in {1, 2, 3, 4, 5} and e to take a value in {256, 384, 512, 768, 1024} and chose optimal values based on log-loss, ignoring configurations that resulted in more than 24GB of GPU memory (which corresponds to the current most high-end consumer GPU, the RTX 3090). An important hyperparameter for CatBoost is the number of boosting iterations. During validation, we determined this quantity using "early stopping". For evaluation runs on a given dataset's test set, we chose the maximum number of iterations required for that dataset's validation runs with optimal hyperparameters. We reproduced r-gcn results from Schlichtkrull et al. [15] using an external implementation [30], as the original code uses deprecated libraries. For the small-scale KGs, we used the hyper-parameter configuration reported in their study [15], but we additionally used early stopping with 10 epochs patience using a validation set held out from the training data to determine the optimal number of layers embedding size  Table 5. The average accuracy and standard error of our 10 measurements. number of epochs. For the larger-scale KGs, we use the hyper-parameters as reported in the "kgbench" paper [34]. All "kgbench" measurements were performed on CPU, to accommodate the high memory requirements for backpropagation. Only for the DBLP dataset, for which no results were reported in the original paper due to high memory requirements, we had to use an embedding size of 10 (as opposed to 16) with 40 base functions to make it fit in our 64GB of CPU RAM. We used an early stopping mechanism, with 10 epochs patience, using the provided validation sets.
We evaluated the r-gcn and rr-gcn setups ten times with different seeds and measured the according test accuracies. We report results for both the rrgcn as described in Algorithm 1 (rr-gcn-ppv) and a version that does not include the ppv features (rr-gcn). The mean accuracy results, and their corresponding standard errors are provided in Table 5.
Link Prediction As in the original work on r-gcns [15], we generated false triples using "negative sampling", where the head or tail of each true triple in  Table 6. Link prediction results for FB15k-237 of r-gcn and our proposed rr-gcn.
E tr is randomly corrupted, i.e., replaced by another entitiy. We trained a scoring function (or "decoder") using the binary cross-entropy loss to score true triples higher than negatives. Whereas r-gcns use the bilinear DistMult decoder [15,10] for link prediction, we preceded DistMult by a small three-layer neural network with hidden and output size 2048, to allow the decoder to transform our taskagnostic embeddings into a space in which the bilinearity holds. As trained r-gcns require much larger hidden sizes for link prediction than for node classification (500 instead of 16 in the original work), our random unsupervised embeddings need to be very large to capture a similar amount of information. As such, we used an rr-gcn-ppv with embedding size 32,000. Using principal component analysis (after feature normalisation), we reduced the representation dimensionality to 8,192 before feeding them to a three-layer preprocessing neural network that was trained jointly with the 2048-dimensional DistMult decoder. We determined the number of epochs to train the decoder using the provided validation set.
Since we used the same final decoding assumption, and training r-gcns for link prediction can take several days [30], for the end-to-end model, we reused the results reported in [15]. The filtered mean reciprocal rank (FMRR) and the hits at 1, 3 and 10 are listed Table 6.
Qualitative Evaluation We perform a qualitative evaluation of our produced rr-gcn embeddings by visualising rr-gcn representations for countries in a small subset of DBpedia [38,39] to gauge if the resulting embedding space makes semantic sense. Figure 1 shows a plot of t-SNE-transformed [40] embeddings generated by a 5-layer rr-gcn-ppv with embedding size 512. Table 5 shows that, for most small-scale datasets, r-gcns score better. However, the differences are not that drastic considering one method is trained end-toend and another is unsupervised and based on random transformations. ppv features, which measure the "representation diversity" around every node, seem to have a positive impact for most datasets. With ppv, our random rr-gcns even score significantly better for the MUTAG dataset. Only for BGS, ppv seemingly detoriates performance. However, the difference between rr-gcn-ppv and rrgcn is statistically insignificant (Mann-Whitney test with cutoff 0.05).

Discussion
For the larger datasets, the average performance of r-gcns and rr-gcns is even more similar, with most datasets statistically significantly favouring rr-  gcn-ppv over a trained model. The difference for MDGENRE, the only "kgbench" dataset where r-gcns score better on average, is statistically insignificant.
The KGs on which rr-gcn performs significantly worse than r-gcn often contain many low-degree nodes (including literals, which usually have only one neighbour) close to the nodes to be classified. We hypothesise that these lowdegree nodes add noise to our representations, which can overpower the useful signal in some datasets. As r-gcns can learn initial node embeddings and per relation transformations, they can learn to ignore unnecessary information, or at least make the useful signal more influential in the final representations. To test this hypothesis, we apply r-gcn and rr-gcn-ppv to filtered versions of AIFB, BGS and AM. In these filtered KGs, low-degree nodes (degree ≤ 5) are removed.
As shown in Table 7, removing low-degree nodes seems to close the gap in average performance for most datasets: the performance for r-gcn stays roughly the same, while rr-gcn results improve. The performance differences to the trained counterparts for AIFB and BGS even become statistically insignificant. For AM, however, low-degree node removal seems to have little to no effect. As we suspect that, for this dataset, the r-gcn learns to ignore a subset of relation types, we inspect the mean absolute values of a trained r-gcn's learned per-relation transformation matrices (true and added inverse relation types are averaged), averaged over all layers. Indeed, Figure 2 illustrates that there are quite a few relation types that a trained r-gcn learns to attenuate. If we take this "relation importance" information into account by removing (in addition to low-degree nodes) all relation types that have less than 60% of the maximum "importance", our rr-gcn obtains an accuracy of 91.31 ± 0.24%, which is even statistically significantly better than trained r-gcns. Table 6 illustrates that link prediction performance for random and trained r-gcns is similar across all evaluation metrics. Which indicates that, even when r-gcns need large hidden sizes to perform downstream tasks (500 in this case), random message passing can capture a comparable amount of signal, without supervision. As only the decoder needs to be trained, the rr-gcn encoder needs to run only once. This results in a significant speedup compared to end-to-end trained r-gcns, which have to perform both a forward and a backward pass trough the entire network for every training step. Consequently, the rr-gcn link prediction model trains in just under two hours, while r-gcns for this task can take days to converge [30]. Figure 1 shows that rr-gcn embeddings preserve semantic relationships between entities: there is a quite clear separation between continents, neighbouring countries are generally close to each other, and some countries with intertwined geopolitical histories even form small clusters.

Conclusion
Inspired by the success of random non-linear transformations in the time series domain, we set out to evaluate random convolutions (rr-gcn) on KGs. In this exploratory study, empirical results show that random transformations usually match or exceed the performance of end-to-end trained r-gcns. Our experiments indicate that a KG's structure alone, which is the only thing our rr-gcns can capture, contains enough semantic information as is, and that when r-gcns do perform better, they mostly learn to ignore certain parts of that structure.
Aside from surprisingly good performance, rr-gcns exhibit other interesting properties: (i) embeddings can easily be generated for a small subset of the graph for testing purposes (whereas r-gcns need to process the entire graph to train their weights), and (ii), as training through backpropagation is not necessary, they can be used to test the influence of new e.g. aggregation functions (such as ppv) before considering differentiable variants. This opens up many new interesting avenues for further research, both in improving the rr-gcns themselves and KG embedding in general, which we discuss subsequently.

Future Work
Apart from being used as a baseline that any trained method should be able to beat, we envision that rr-gcns might be helpful as a tool for improving and/or developing other embedding techniques. By further investigating in which situations or for which substructures random information aggregations are insufficient, trained algorithms could be altered to better target problematic computations. Hybrid networks with both trained and random weights could offer the best of both worlds.
In this work we explored information aggregation using random transformation matrices. Other strategies, e.g. the integration of algebraic embedding priors into message passing [16,18], could be randomised as well.
As it gives a measure of the representation diversity around a node and thus captures more of the KG's structure, ppv seems like promising addition to the random message passing toolbox. However, the independent computation of ppv and regular features is somewhat arbitrary. Future work could research other options such as calculating ppv features only after the last message passing layer, or even outright replacing the averaging used in r-gcns and using ppv as the aggregation. Moreover, differentiable alternatives to ppv could be researched to improve trained r-gcns. Future research might reveal new features that capture other aspects of graph, to further improve performance.
Even though for some datasets (e.g. DBLP), rr-gcns require less memory than their trained counterparts for comparable performance, the memory requirements are still substantial for graphs with many nodes. Future research could trade in some of these memory requirements for compute by e.g. aggregating many small node embeddings.
Supplemental Material Statement: Source code for the rr-gcn embedder and all experiments is attached with the submission on EasyChair and, if accepted, will be published on GitHub.