{'fraction_non_alphanumeric': 0.039503075181298, 'fraction_numerical': 0.010587191334414492, 'mean_word_length': 4.581895815542271, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Graph Sequential Neural ODE Process for Link Prediction on Dynamic and Sparse Graphs

Link prediction on dynamic graphs is an important task in graph mining. Existing approaches based on dynamic graph neural networks (DGNNs) typically require a significant amount of historical data (interactions over time), which is not always available in practice. The missing links over time, which is a common phenomenon in graph data, further aggravates the issue and thus creates extremely sparse and dynamic graphs. To address this problem, we propose a novel method based on the neural process, called Graph Sequential Neural ODE Process (GSNOP). Specifically, GSNOP combines the advantage of the neural process and neural ordinary differential equation that models the link prediction on dynamic graphs as a dynamic-changing stochastic process. By defining a distribution over functions, GSNOP introduces the uncertainty into the predictions, making it generalize to more situations instead of overfitting to the sparse data. GSNOP is also agnostic to model structures that can be integrated with any DGNN to consider the chronological and geometrical information for link prediction. Extensive experiments on three dynamic graph datasets show that GSNOP can significantly improve the performance of existing DGNNs and outperform other neural process variants.


INTRODUCTION
Graphs are ubiquitous data structures in modeling the interactions between objects. As a fundamental task in graph data mining, link prediction has been found in many real-world applications, such as social media analysis [15], drug discovery [1], and recommender systems [9]. However, data in these applications is often dynamic. For example, people interact with their friends on social media every day, and customers contiguously click or buy products in a recommender system. Therefore, it is essential to consider the nature of dynamics for accurate link prediction. As shown in Figure  1, link prediction on dynamic graphs, where interactions arrive over time, aims to predict links in the future by using historical interactions [16,19,31].
Recently, graph neural networks (GNNs) have shown great capacity in mining graph data [21,23,24,35,37,41]. Under the umbrella of dynamic graph neural networks (DGNNs), several approaches have recently been proposed for predicting links on dynamic graphs [29,33,34]. As shown in the left panel of Figure 1, these methods typically require a large amount of historical links for training, which is unfortunately not always available [39,42]. For instance, a new-start online shopping platform could only have few days' user-item interactions. DGNNs trained on such limited data would be easily overfitting and cannot generalize to new links. In contrast, the platform expects models to accurately predict future interactions as early as possible, which leaves little time to collect the data. Furthermore, because the links in dynamic graphs arrive over time, the limitation of historical data in dynamic graphs would be exacerbated by the missing of interactions, resulting in extremely sparse graphs. Thus, how to enable effective link prediction on dynamic and sparse graphs (e.g., with limited historical links, as shown in the right panel of Figure 1) remains a significant challenge in this area.  Neural process (NP) [12], a new family of methods, opens up a new door to dealing with limited data points in machine learning [25,26,32]. Based on the stochastic process, NP first draws a distribution of functions from the data by using an aggregator, and then it samples a function from the learned distribution for predictions. By defining the distribution, NP can not only readily adapt to newly observed data, but also estimate the uncertainty over the predictions, which makes models generalize to more situations, rather than overfitting to the sparse data. NP has already been applied in many applications, such as 3D rendering [10], cold-start recommendation [22], and link prediction on static graphs [20].
However, NP and its variants [4,17,32] cannot be directly applied to the link prediction on the dynamic graphs. Vanilla NP [12] draws the distribution using a simple average-pooling aggregator, which ignores the sequential and structural dependence between nodes for accurate link prediction [18,27,34]. For example, as shown in the Figure 2a, node and respectively have an interaction with node at time 1 . Therefore, node and are quite likely to interact in the future 2 . In this case, simply using average-pooling aggregator would erase the dependence and lead to a sub-optimal result. SNP [32] adopts a RNN aggregator to consider the sequential information, but it still fails to consider the derivative of the underlying distribution and uses a static distribution through time. For instance, as shown in Figure 2b, the links in dynamic graphs arrive irregularly [5,13,36]. Additionally, an important event could lead to a large number of links occurring in a short time, which is shown by the spikes in Figure 2b. Therefore, using a static distribution drawn from the historical data is inappropriate.
In this paper, we introduce a novel class of NP, called Graph Sequential Neural ODE Process (GSNOP) for link prediction on dynamic graphs. Specifically, GSNOP first draws a distribution of link prediction functions from historical data using a dynamic graph neural network encoder, which can be incorporated with any existing DGNN model to capture the chronological and geometrical information of dynamic graphs. Second, we propose a sequential ODE aggregator, in which we consider the function of dynamic graph link prediction as a sequence of dynamic-changing stochastic processes, and we adopt the RNN model to specify the dependence between data. To handle the irregular data, we apply the neural ordinary differential equation (ODE) [7] which is a continuoustime model defining the derivative of the hidden state with a neural network. Neural ODE is used to model the derivative of the neural process and infer the distribution in the future. In this way, we can obtain the distribution of link predict functions at any contiguous timestamp. Finally, we sample the function from the distribution for link prediction, which introduces the uncertainty over prediction results, offering a resilient way to deal with the sparse and limited data.
The main contributions of this paper are summarized as follows: • We propose a novel class of NP, called Graph Sequential Neural ODE Process (GSNOP). To the best of our knowledge, this is the first work to apply NP for link prediction on dynamic and sparse graphs. • We propose a dynamic graph neural network encoder and a sequential ODE aggregator, which inherits the merits of neural process and neural ODE to model the dynamic-changing stochastic process. • We conduct extensive experiments on three public dynamic graph datasets. Experiment results show that GSNOP can significantly improve the performance of existing DGNNs on dynamic and sparse graphs.

RELATED WORK
In this section, we first review some representative DGNN models and then introduce the NP and its variants.

Dynamic Graph Neural Networks
Dynamic graph neural networks (DGNNs) can be roughly grouped into two categories based on the type of dynamic graphs: discretetime dynamic graphs (DTDG) and continuous-time dynamic graphs (CTDG). DTDG-based methods model dynamic graphs as a sequence of graph snapshots. DySAT [31] first generates node embeddings in each snapshot using an attention-based GNN, then summarize the embeddings with a Temporal Self-Attention. EvloveGCN [28] first trains a GNN model on each snapshot and adopts the RNN to evolve the parameters of GNN to capture the temporal dynamics. Continuous-time dynamic graph (CTDG) is a more general case where links arrive with contiguous timestamps. JODIE [19] employs two recurrent neural networks to update the node embedding by message passing. TGAT [38] proposes a temporal graph attention layer to aggregate temporal-topological neighbors. Furthermore, TGN [29] introduces a memory module to incorporate the historical interactions. APAN [33] decouples the message passing and memory updating process, which greatly improves the computational speed in CTDG. However, these DGNNs require sufficient training data, which largely limits their applications. MetaDyGNN [39] adopts the meta-learning framework for few-shot link prediction on dynamic graphs. But it requires new observed links to finetune the model, which shares different settings with our work.

Neural Process
Conditional neural process (CNP) [11] is the first member of the NP family. CNP encodes the data into a deterministic hidden variable that parametrizes the function, which does not introduce any uncertainty. To address the limitation of CNP, neural process (NP) [12] is a stochastic process that learns a latent variable to model an underlying distribution over functions, from which we can sample a function for downstream tasks. ANP [17] marries the merits of CNP and NP by incorporating the deterministic and stochastic paths in an attentive way. However, ANP could be memory-consuming when processing a long sequence, since it needs to perform selfattention on all the data points. SNP [32] further adopts RNN to consider the sequential information by modeling the changing of distribution. But it fixes the distribution in testing due to the lack of new observed data. NDP [26] applies the neural ODE [7] to the decoder of neural process to consider the dynamic between data. NPs have also been applied to address many problems, such as modeling stochastic physics fields [14], node classification [4], and link prediction on static graphs [20]. However, none of them applies NP for dynamic graph link prediction. Besides, existing NPs cannot be directly used in dynamic graphs as they ignore the sequential and structural dependence as well as the irregular data.

PROBLEM DEFINITION AND PRELIMINARY
In this section, we formally define our problem and introduce some key concepts used in this paper.

Problem Definition
Dynamic graphs can be grouped into two categories: discrete-time dynamic graphs (DTDGs) and continuous-time dynamic graphs (CT-DGs). A DTDG can be defined as a sequence of graph snapshots separated by time interval: is a graph taken at a discrete timestamp . A CTDG is a more general case of a dynamic graph, which is constituted as a sequence of links

Neural Ordinary Differential Equation.
Neural ordinary differential equation (ODE) [7] is a continuous-time model which defines the derivative of the hidden state with a neural network. This can be formulated as where ( ) denotes the hidden state of a dynamic system at time , and denotes a neural network that describe the derivative of the hidden state w.r.t. time. Thus, the neural ODE can output the ( ) at any timestamp by solving the following equation Equation 2 can be solved by using numerical ODE solver, such as Euler and fourth-order Runge-Kutta. Thus, it can be simplified as 3.2.2 Neural Process. Neural process (NP) [11,12] is a stochastic process that learns an underlying distribution of the prediction function : → with limited training data. Specifically, NP defines a latent variable model using a set of context data = {( , )} =1 to Encoder Encoder

Context Data Target Data
Decoder Aggregator Aggregator Generation Inference Figure 3: The framework of neural process. Circles denote the input and output data; round squares denote the latent variables.
model a prior distribution ( | ). Then, the prediction likelihood on the target data = {( , )} + = is modeled as We illustrate the framework of NP in Figure 3. The aim of NP is to infer the target distribution from context data. NP first draws a distribution of from the context data , written as ( | ). By using an encoder, NP learns a low-dimension vector for each pair ( , ) in the context data. Then, it adopts an aggregator to synthesize all the into a global representation r, which parametrizes the distribution, formulated as ∼ N (r), (r) . Last, it samples a from the distribution which is fed together with the ∈ into a decoder to predict the label for target data, which is written as ( : + | : + , ). Because the true posterior is intractable, the NP is trained via variational approximation by maximizing the evidence lower bound (ELBO) as follows ( : where ( | , ) approximates the true posterior distribution also parameterized by a neural network.

GRAPH SEQUENTIAL NEURAL ODE PROCESS
In this section, we introduce the proposed Graph Sequential Neural ODE Process (GSNOP), which combines the merits of neural process and neural ODE to model the dynamic-changing stochastic process. GSNOP can be integrated with any dynamic graph neural network (DGNN) to simultaneously consider the chronological and geometrical information for dynamic graph link prediction. The framework of GSNOP is illustrated in Figure 4.

Setup and Framework
Given a CTDG G, we adopt GSNOP to infer the distribution of an underlying link prediction function : ( ( )) → , ∈ {0, 1} from context data. By specifying a timestamp , we split G into the context data G ≤ and target data G > . Each ( ) ∈ G ≤ occurred prior to , and ( ) ∈ G > arrives when > . At testing time, we treat all the historical interactions as context data, and predict links in the future. GSNOP follows the framework of NP, thus, the encoder, aggregator, and decoder in GSNOP are defined as follows.
As shown in Figure 4, GSNOP first draws a historical distribution of from the context data G ≤ by using a dynamic graph neural network encoder, written as ( |G ≤ ). GSNOP adopts the dynamic graph neural network to learn a low-dimension vector for each link in the context data. Then, it adopts an sequential ODE aggregator to synthesize all the into a global representation r , which parametrizes the distribution ( |G ≤ ), formulated as ∼ N (r ), (r ) . Following, GSNOP applies the neural ODE on the ( |G ≤ ) to infer the distribution in the future > , written as ( > |G ≤ ) Last, GSNOP sampled a > from the future distribution which is fed together with the ( ) ∈ G > into a link prediction decoder to predict the links in target data, which is formulated as ( ( ), > ) → . Each component is introduced in detail as follows.

Dynamic Graph Neural Network Encoder
The encoder is applied to each pair ( ( ), ) in the context data, where denotes the existence of the links. It tries to capture the connection between ( ) and to infer the distribution of link prediction function . Link prediction on dynamic graph has been studied for years. Previous research has shown that the dynamic graph neural networks (DGNNs) have great potential to capture the chronological and geometrical information for dynamic graph link prediction [29,38]. Therefore, we adopt the DGNNs as the encoder of GSNOP.
Given a pair ( ( ), ), we first adopt a DGNN to generate the representation ℎ of nodes , ∈ ( ), which is formulated as where ℎ −1 denotes the representation of node in the last timestamp, < ( ) denotes a set of nodes that interacted with in previous timestamps, and (·) denotes the aggregation operation defined by DGNN. Equation 6 can be written as DGNN( , < ( )) for simplification. We can use arbitrary DGNN to consider the dynamic and graph structure information for a given node.
Then, we use a MLP to encode each pair ( ( ), ) into a representation , which is formulated as where || denotes the concatenation operation, denotes the existence of the link between and in the context data, and denotes the time embedding [38] sharing the same dimension with that can be learned during the training.

Sequential ODE Aggregator
The aggregator summarizes encoder outputs into a global representation r that parametrizes latent distribution ∼ N (r ), (r ) . Originally, NP [7] adopts a simple average-pooling aggregator, which is formulated as However, it fails to consider the sequential and structural dependence between nodes in dynamic graphs. Vanilla NP assumes that data in the context data is i.i.d., thus it adopts a simple average-pooling aggregator to define the joint distribution ( |G ≤ ). However, in dynamic graphs, we need to consider the sequential and structural dependence between nodes for accurate link prediction [18,27,34]. In this case, simply using average-pooling aggregator would erase the dependence between ( ) ∈ G ≤ and lead to a sub-optimal result. To consider the sequential and structural dependence, inspired by Sequential Neural Process (SNP) [32], we consider the dynamic graph link prediction function as a sequence of dynamic-changing stochastic processes {P 1 , . . . , P }. At each timestamp , we can observe a set of context data {( ( ), ) ∈ G }, where G might differ over time. We can infer a distribution P by using all the historical data G ≤ , which can be formulated as Because, P −1 denotes the distribution drawn from the G < . By modeling the changing of process P −1 → P , we can consider the dependence among context data. Thus, we adopt a RNN model to specify the state transition model in Eq. (9), which is formulated as where r 1 is computed by averaging 1 generated from G 1 . By using the RNN aggregator, we can obtain the r from the representation of context data { 1 , . . . , } and define the distribution ( |G ≤ ). But, it still fails to consider the derivative of the underlying distribution.
In vanilla NP, it infers the distribution of link prediction function ( |G ≤ ) from historical context data, and fixes it for future link prediction. However, in dynamic graphs, the links arrive in different frequencies [5,13,36]. Therefore, it is inappropriate to use a static distribution over time, although we can incorporate the new-coming links into the context data and update the distribution by using Eq. 10. In most cases, it is infeasible to obtain the links in the future. As a result, we need to consider the derivative of distribution over time to deal with irregular data.
Recently, neural ordinary differential equation (ODE) [7] has shown great power in dealing irregular time series data [30]. Thus, in GSNOP, we apply the neural ODE to model the derivative of neural process and infer the distribution in the future ( > |G ≤ ). Given a distribution ( |G ≤ ) drawn from context data G ≤ , it is parameterized by the global latent variable r . We apply the neural ODE on the r to obtain the latent variable at any future time > = + △ , which is formulated as where denotes the neural ODE function, which is formulated as (r , ) = ℎ( (r + )).
We use the ℎ as the activation function to keep from gradient exploration.
After obtaining the > , the distribution parameterized by r > can be considered as a Gaussian distribution N (r > ), (r > ) , where the mean (r > ) and variance (r > ) are formulated as

Link Prediction Decoder
The decoder is considered as the underlying link prediction function ( ( ), ) → . Given two nodes and at time to be predicted, can be formulated as whereˆdenotes the prediction result.
The decoder is conditioned on the variable > sampled from the distribution. This injects the uncertainty into the prediction results, making GSNOP generalize to more situations, even with limited data. In addition, as more data observed, the distribution could be refined and provide more accurate predictions, which is demonstrated in section 5.5.

Optimization
Given the context data G ≤ and target data G > , the object of GSNOP is to infer the distribution ( > |G ≤ ) from the context data that minimize the prediction loss on the target data. Therefore, the generation process of GSNOP can be written as where is the link prediction decoder, and E = { ( ) ∈ G > } denotes the links to be predicted in the target data.
Since the decoder is non-linear, it can be trained using an amortised variational inference procedure. We can optimize the evidence lower bound (ELBO) to minimize the prediction loss on target data given the context data ( |E , G ≤ ), which can be derivate as where ( > ) denotes the true posterior distribution of > . Thus, we can obtain the final ELBO loss as where ( |E , > ) = ( ) ∈E ( | ( ), > ) denotes the prediction loss on the target data and is parameterized by the decoder. The ( > ) is approximated by the variational posterior ( > |G ≤ , G > ) calculated by the encoder and RNN aggregator, formulated as The prior ( > |G ≤ ) is approximated by the neural ODE in Eq. 11, written as ( > |G ≤ ). By minimizing the KL divergence between variational posterior and the prior ( > |G ≤ ), it encourages the neural ODE to learn the derivative of distribution and infer the distribution in the future. Extended derivation of Eq. 26 can be found in the supplementary file of this link 2 .
Noticeably, unlike neural ODE process [26] which applies neural ODE on the decoder, GSNOP applies neural ODE to infer the distribution of function at any contiguous timestamp. This is more suitable for the dynamic-changing stochastic process modeled for dynamic graph link prediction.
Testing. At test time, given a set of historical interactions G ≤ , we treat them as the context data and draw a global representation r using the DGNN encoder as well as RNN aggregator. Then, given a link ( + △ ) to be predicted, we apply the neural ODE to infer the distribution at future, formulated as Last, we use the decoder to predict the link ( ( + △ ), > ) → . Given a CTDG containing links in context data and links to be predicted in the target data, the time complexity of GSNOP is ( + ) since model need to encode each link in the context data and predict each target link. This shows that GSNOP scales well with the size of the input. At testing time, we also batch the samples by using the dummy variable trick [6] to speed up the integration in neural ODE.

EXPERIMENT 5.1 Datasets
We conduct experiments on three widely used CTDG datasets: WIKI [29], REDDIT [29], and MOOC [19]. Following the settings of the previous work [38], we focus on the task of predicting links in the future using limited historical interactions. Intuitively, with the links coming over time, the CDTG would become denser. To simulate different levels of graph sparsity, we split the CDTG chronologically by timestamps in the entire duration with two ratios, 30%-20%-50% and 10%-10%-80% for training, validation, and testing. The datasets with different splits are labeled by their training ratio (e.g., WIKI_0.3 and WIKI_0.1). The sparsity of graphs can be quantified by the density score, calculated as 2| E | | V | ( | V |−1) , where |E | and |V | denote the number of links and nodes in the training set. The statistics of the datasets is shown in Table 1.

Baselines
Since GSNOP is agnostic to model structure, we choose several stateof-the-art DGNN as baselines (i.e., JODIE [19], DySAT [31], TGAT [38], TGN [29], and APAN [33]). Detailed introductions of these DGNNs can be found in section 2.1. To demonstrate the effectiveness of GSNOP, we adopt these DGNNs respectively as the encoder in GSNOP to illustrate the performance improved by GSNOP. In addition, we also select other neural process variants (i.e., Neural 1 Due to the lack of link features, we randomly generated link features for MOOC. Process (NP) [12], Conditional Neural Process [11] and Sequential Neural Process (SNP) [32]) to demonstrate the superiority of GSNOP.

Settings
In experiments, we adopt the average precision (AP) and mean reciprocal rank (MRR) as evaluation metrics. To fairly evaluate the performance in sparse settings [40], we randomly select 50 negative links for each positive link in the evaluation stage. For each DGNN baseline, we set the layer of the GNN to 2 and the number of neighbors per layer to 10. The memory size for APAN is set to 10, while 1 for JODIE and TGN. For snapshot-based DySAT, we set the number of snapshots to 3 with the duration of each snapshot to be 10,000 seconds. For all methods, the dropout rate is swept from {0.1,0.2,0.3,0.4,0.5}, and the dimension of output node embeddings is set to 100. We use the function in section 4.4 to predict links.
For GSNOP, the dimension of , , and are all set to 256. The Monte Carlo sampling size for approximating E ( > | ≤ , > ) in L is set to 10. We use GRU [8] as the RNN aggregator and choose 5 order Runge-Kutta of Dormand-Prince-Shampine [3] as the ODE solver with relative tolerance set to 1 − 5 and absolute tolerance set to 1 − 7. We use Adam as the optimizer, and the learning rate is set to 0.00001. The implementation of GSNOP is available at this link 2 .

Performance Comparison
In this section, we evaluate the performance of GSNOP on different datasets. The results are shown in Table 2. From the results, we can see that GSNOP enables to improve the performance of all the baselines with respect to different datasets and training ratios.
Specifically, when the historical training data is sufficient, all DGNNs achieve relatively good performance in WIKI_0.3, RED-DIT_0.3, and MOOC_0.3. However, when training data is limited, the performance of some baselines drops significantly (e.g., JODIE, TGAT, and APAN). The possible reason is that these DGNNs use historical data to generate node embeddings. When data is limited, the quality of the generated embeddings cannot be guaranteed. Thus, it is inaccurate to use such embeddings for link prediction directly. In addition, the data distribution of the testing set could be diverse from the training set. This would lead the model overfit to the historical data and cannot generalize to the future links. The performances between MOOC_0.3 and MOOC_0.1 are similar. This is because MOOC is a denser CTDG with short duration. Thus, most DGNNs perform well in this dataset.
In GSNOP, we model the distribution of function from the historical data by learning a latent variable. This variable introduces a global view of knowledge, from which we can sample a link prediction function rather than only use the node embeddings. Besides, the distribution introduces uncertainty over prediction results. In this way, GSNOP can generalize to more situations instead of overfitting to the training data. Last, by adopting the neural ODE, GSNOP enables to infer the distribution in the future with a small amount of historical data. Noticeably, the improvement of GSNOP is sometimes limited (e.g., results in MOOC_0.1). The possible reason is      that GSNOP learns the latent variable by optimizing the evidence lower bound. This does not always result in a meaningful latent variable that parametrizes the distribution [2,25]. We will leave this for future exploration.

Performance at Different Levels of Sparsity
In the real-world, we might only observe partial links for an underlying CTDG, which aggravates the sparsity. To further demonstrate the performance improvement under different sparsity, we sample subsets of training links from the original training set with various sample ratios, while leave the validation and test set unchanged. For instance, the 30% sample ratio in WIKI_0.3 denotes that we randomly mask 70% of links in the training set and use the remaining 30% links for training. The density score of the sampled dataset can be calculated by multiplying the sample ratio and the density of the original dataset. We sweep the sample ratio from 100% to 10%. The results are shown in Table 3.
From the results, we can see that GSNOP outperforms the baselines under different levels of sparsity. Specifically, with the sample ratio decreasing, the performance of each DGNN also decreases. This is because when the training data is sparse, the model cannot capture enough interactions for link prediction. However, with GSNOP, we can define the distribution with limited interactions to estimate the underlying graph. Thus, it can better handle the sparsity situation. In addition, GSNOP can incorporate new observations to enhance the distribution. With the sample ratio increasing from 10%-100%, the improvement bought by GSNOP increases. The reason is that by adding new observed links into the latent variable, GSNOP enables to more accurately define the distribution of function with the help of DGNN encoder and RNN aggregator.

Ablation Study
In this section, we compare our GSNOP with other neural process variants (i.e., vanilla neural process (NP) [12], conditional neural process [11] and sequential neural process (SNP) [32]) to show the superiority of our method in dynamic graph link prediction. For each variant, we use our dynamic graph neural network encoder and link prediction decoder but maintain their original way of generating the distribution. The results in Table 4 would help to demonstrate the effectiveness of our sequential ODE aggregator.  SNP  GSNOP  origin  NP  CNP  SNP  GSNOP   AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP  MRR  AP   From the results, we can see that GSNOP performs better than other neural process variants. NP is the original version of neural process. Since it ignores the sequential and structural dependence, NP has the minimum improvement. Compared with other variants, CNP achieves the worst performance. The reason is that CNP is a deterministic version of neural process. Instead of learning the latent variable, CNP directly models the distribution conditioned on the context data, which could be inaccurate when the context data is limited. SNP adopts the RNN as the aggregator, which considers the sequential dependence. However, it fails to capture the distribution derivative, making SNP unable to handle irregular data. GSNOP incorporates both the RNN aggregator and neural ODE, which simultaneously exploits the sequential information and enables inferring the distribution at any timestamps. Thus, GSNOP achieves the best results among all the variants.
To further demonstrate the effectiveness of Sequential ODE aggregator, we split links in the test set of WIKI_0.1 chronologically into four groups based on their timestamps. For example, the group [0, 0.25) denotes the links between 0 and 0.25 * ( ) in the test set. The prediction loss of each group with DySAT as the encoder is shown in Figure 5a. From the results, we can see that Origin and CNP have relatively high losses across different groups. Due to the ignoring of sequential dependence, NP cannot infer the distribution in the future. Thus, the loss of NP increases with the timestamps. Despite SNP capturing the sequential information, it still fails to address the irregular data. As shown in Figure 5b, there is a high spike in group [0.25, 0.5), and the arriving links are decreasing in group [0.75, 1.00). Thus, the losses of SNP are higher in these irregular groups. With the help of the sequential ODE aggregator, GSNOP models the distribution derivative and maintain relatively low losses across different timestamps.

Parameters Analysis
We investigate two critical hyperparameters (i.e., learning rate and the dimension of ) in Figure 6. From the results, we can see that the learning rate could severely impact the performance of GSNOP. This is because GSNOP adopts the dummy variable trick [6] to speed up the neural ODE. However, this will incur the gradient explosion problem when integrating in a large time period. Thus, we need to select a relatively small learning rate (i.e., 1 −5) to avoid gradient explosion. However, a too small learning rate also causes underfitting and leads the performance drops. With the dimension of increasing, the performance of GSNOP improves and reaches the best at 256. The reason is that a small dimension of cannot well model the stochastic of distribution, but an over large dimension would also introduce noises when optimizing the evidence lower bound, which deteriorates the performance.

CONCLUSION
In this paper, we propose a novel NP named GSNOP for link prediction on dynamic graphs. We first specify a dynamic graph neural network encoder to capture the chronological and geometrical information of dynamic graphs. Then, we propose a sequential ODE aggregator, which not only considers the sequential dependence among data but also learns the derivate of the neural process. In this way, it can better draw the distribution from limited historical links. We conduct extensive experiments on three public datasets, and the results show that GSNOP can significantly improve the link prediction performance of DGNNs in dynamic and sparse graphs. Experiment results comparing with other NP variants further demonstrate the effectiveness of our method.

ACKNOWLEDGE
This work is supported by an ARC Future Fellowship (No. FT21010 0097).