{'fraction_non_alphanumeric': 0.031482601720102046, 'fraction_numerical': 0.00970516293627206, 'mean_word_length': 4.991490702804916, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Temporal Graph Network Embedding with Causal Anonymous Walks Representations

Many tasks in graph machine learning, such as link prediction and node classification, are typically solved by using representation learning, in which each node or edge in the network is encoded via an embedding. Though there exists a lot of network embeddings for static graphs, the task becomes much more complicated when the dynamic (i.e. temporal) network is analyzed. In this paper, we propose a novel approach for dynamic network representation learning based on Temporal Graph Network by using a highly custom message generating function by extracting Causal Anonymous Walks. For evaluation, we provide a benchmark pipeline for the evaluation of temporal network embeddings. This work provides the first comprehensive comparison framework for temporal network representation learning in every available setting for graph machine learning problems involving node classification and link prediction. The proposed model outperforms state-of-the-art baseline models. The work also justifies the difference between them based on evaluation in various transductive/inductive edge/node classification tasks. In addition, we show the applicability and superior performance of our model in the real-world downstream graph machine learning task provided by one of the top European banks, involving credit scoring based on transaction data.


INTRODUCTION
It is crucial for banks to be able to predict possible future interactions between companies: knowing that one company will be a client of another one gives an opportunity to offer financial and other siervices. It is also important to have comprehensive and meaningful information about each client. If this knowledge is expressed as client embeddings, then the problem of their compactness and expressiveness emerges. Banks own large datasets of financial interactions between their clients, which can be used for training and testing models solving link prediction.
Graph structures that describe dependencies between data are nowadays widely used to improve effectiveness of machine learning models trained on streaming data. In order to use conventional machine learning frameworks, it is necessary to develop a vector representation of the graph (such as network embeddings) by combining attributes from nodes (labels, text, etc.) and edges (i.e. weights, labels, timing) and taking into account the dynamic graph structure appearing in real-world problems. Existing graph embedding models have been actively studied in recent years in attempts to apply deep learning methods for network representation learning. Hundreds of graph embeddings models have been developed in computer vision, text processing, recommendation systems, and interdisciplinary research in biology and genetics [19]. All approaches are united by a common problem statement, which is to learn a model for the selected type of networks and important graph statistics. This will make it possible to apply standard machine learning frameworks and at the same time generalize attribute and structural information from the data.
Recently, modern machine learning methods have been proposed for processing networks and solving various prediction tasks on the arXiv:2108.08754v2 [cs.LG] 24 Aug 2021 local level. [25,35,39,42]. Examples include node classification and link prediction, for nodes both seen and unseen during the process of model training. In practice, they represent important problems to be solved on large dynamic network data, for e.g., transaction data between companies and bank clients can be used to predict future transactions; user search history on the Web can be used to generate contextual advertising instances; disease transmission data can be used to predict epidemics dynamics.
Although most of the previous works on graph representation learning mainly focus on static graphs (with a fixed set of nodes and edges), there are many real-world applications in which graphs evolve over time, like social networks or sales data. One particularly common sub-type of graphs used to represent such structures is a temporal graph. It is a graph in which each edge has a time index, indicating a moment in time when the interaction, represented by an edge, occurred.
There are various problems when switching from static to dynamic networks [46], among which computational complexity and variance of connectivity patterns over time. In addition, for practical applications, one needs to have models suitable for inference in inductive settings which enable proper prediction on the fly with rear overall model retraining for large network data [28].
In this work, we describe a novel network embedding which combines the best elements of the efficient Temporal Graph Network embedding (TGN) [12] with fast computed historic node embeddings and highly precise Causal Anonymous Walks (CAW) [61] with attention over temporal edge dynamics. To properly evaluate the proposed model, we present an experimental framework for temporal network embedding evaluation in downstream graph machine learning tasks, allowing integration of various temporal network embeddings and different temporal network data under a unified evaluation framework shown in Fig. 1.
Our main contributions with this work consist of the following: (1) Novel temporal network embedding model achieving stateof-the-art results in various temporal graph machine learning tasks; (2) Standardized temporal network embedding evaluation framework and comparison of state-of-the-art models under common training setting, providing new insights and clarification of real-world performance compared to reported in the original research articles.
In addition, we prove the effectiveness of the proposed pipeline and its sub-modules via extensive ablation study and provide the industrial application of the proposed approach, involving the transaction data of a major European bank. We showed that feature enrichment of temporal attention over temporal edge random walks improves both quantitative and qualitative results in the real-world application of a banking graph machine learning task.

PRELIMINARIES & PROBLEM STATEMENT
In order to proceed with the problem statement, we describe basic concepts used throughout the text following notations by [28]. We use G(V, E) to denote a static graph, where V is the set of its vertices, and E is the set of edges of the graph, and A is an adjacency matrix of that graph. Dynamic graph in general is a graph, which structure and node/edge attributes change over time.
Generally, events may contain an updated state of the node, but in our experiments we consider all node features to be static, and represent them as X -| | × matrix, where is the node feature dimensionality.
In this work, we will use dynamic graph and temporal graph as interchangeable terms. We outline two possible kinds of dynamic graph representations below. There are two standard views on temporal network representation as a data structure: • Discrete-time dynamic graph (DTDG) or snapshot-like graph is a sequence of snapshots from a dynamic graph, sampled at regularly-spaced times. Event stream may be represented as concatenation of index matrix containing transactions vectors ( ) = ( , , ) with source and target node IDs, timestamp, and temporal edges' features.
We will refer to both CTDG and DTDG as dynamic graphs, although we will focus more on CTDGs as a natural representation of transactions in banking networks, which appear in non-uniform timestamps and represent a real-world streaming structured data, rather than discretized snapshot representation DTDG.
Finally, in our study, we consider such temporal graph machine learning problems as node classification and link prediction, both in transductive and inductive settings: • Transductive edge prediction evaluates whether a transaction between two priorly known nodes occurred at a given time; • Inductive edge prediction predicts a transaction between known and unknown nodes at a given time; • Transductive node classification determines the dynamic label of a priorly known node; • Inductive node classification determines the dynamic label of a priorly unknown node.
All the mentioned above problems on the chosen datasets can be formulated as binary classification (with effortless extension to a multi-class case), which is evaluated via AUC-ROC (area under curve for receiver operating characteristic) and AP (average precision) quality metrics measuring performance for the classification at various error threshold settings.

RELATED WORK
In this section, we overview state-of-the-art methods of constructing network embeddings for static and dynamic networks using taxonomies suggested in [19] and [11]. We focus on dynamic network embedding as an evolutionary process of graph formation.

Static graph embedding methods
When constructing network embeddings via solving the optimization problem, researchers usually focus on three main concepts: matrix factorization, node sequence methods and methods based on deep learning. We consider the snapshot method, in which the current snapshot of a temporal network is taken and missing links are predicted based on the available graph information.
Factorization techniques can be applied to different graph representations and optimized for different objectives, such as direct decomposition of the graph adjacency matrix [30,38,49] or approximating proximity matrix [7,54]. Despite factorizations being widely used in recommendation systems, these models have high computational complexity and are difficult to extend for inductive learning.
Inspired by word2vec [53], sequence-based methods aim to preserve local node neighborhoods based on random walks. The two most prominent examples of models in this class are DeepWalk [6] and Node2vec [1]. Anonymous graphs walks have been proposed in [20]. However, their adaptations to temporal random walks have limited applications, since they require retraining after adding new edges.
Recently, advances in geometric deep learning led to the creation of graph neural networks combining the best out of both fully connected and convolutional neural network architectures. Most of them use the method of neighborhood information aggregation from graph convolution network (GCN) [39] and extend it with classical deep learning architectures such as recurrent neural network (RNN) [25], attention mechanism [42], generative adversarial network (GAN) [35], and graph transformers [63].
Recent studies show that a combination of deep learning models with semi-supervised techniques gives state-of-the-art results in terms of scalability, speed and quality in downstream tasks [19]. However, static models are limited by the necessity to retrain the model with each significant change of graph structure.

Dynamic graph embedding methods
Methods for dynamic graphs are often extensions of those for static ones, with an additional focus on the temporal dimension and update schemes [3]. All these techniques can be categorized according to which model of graph evolution representation is chosen: Continuous-Time Dynamic Graphs (CTDG) or Discrete-Time Dynamic Graphs (DTDG).

DTDG-focused methods.
Most of the early work on dynamic graph learning focuses exclusively on discrete-time dynamic graphs. Such models encode snapshots individually to create an array of embeddings or aggregate the snapshots in order to use a static method on them [55]. All DTDG models can be divided into several categories, according to the approach they use for dealing with the temporal aspect of a dynamic graph.
Single-snapshot models. Static models are used on the graph snapshots to make predictions for the next one ( [31,41,47]). Another way of implementing this methodology, called TI-GCN (Time Interval Graph Convolutional Networks) via residual architectures was proposed in [44,62]. Besides single snapshots, these works use information from networks formation [44] represented by edge differences of several snapshots.
Multi-snapshot models. In [36,52], authors learn structural information of each snapshot by separate models. Authors of [36] compute individual sets of random walks for each snapshot in Node2Vec fashion and learn final node embeddings jointly, while in [52] autoencoders for each snapshot were trained in a consistent way to preserve similarity between consequent graph updates.
RNN-based models. In contrast to previous methods, models in this category aim to capture sequential temporal dependencies mostly by feeding the output node embeddings or graph structure of each snapshot into RNN. Thus, GCN is combined with long shortterm memory (LSTM) in [13,21,60] or gated recurrent units (GRU) in [34,57]. Following these ideas, authors of [9,48] add residual connections to propagate topological information between neighboring snapshots. Recently, some works [2,27] have proposed to use GANs in combination with RNNs. On the other hand, EvolveGCN [3] argues that directly modeling dynamics of the node representation will hamper the model performance on graphs with dynamic node sets. Instead of treating node features as the input to RNN, it feeds the weights of the GCN into the RNN.
Temporal graph attention. Inspired by advances in natural language processing (NLP), models in this class leverage attentionmechanism to capture temporal information. In [18,24,26], authors follow the RNN module by an attention mechanism to take into account the contribution of its hidden states or leverage self-attention mechanism without RNN stage [4].
Convolutional models. Although previous models can capture sequential dependencies, in practice, most of them use a limited number of historical snapshots. So, convolutions are used to propagate information between snapshots in [8,14,65].
Despite promising results, most of the models struggle from two disadvantages. First, methods lose the order of edge formation and reflect only partial information on network formation [62]. Second, computing static representations on each snapshot is inefficient in terms of memory usage on large graphs [64] and can not be used in practical applications.

CTDG-focused methods.
Continuous-time dynamic graphs require different approaches as it becomes computationally difficult to work with the entirety of such graphs after each interaction [43]. Below we provide a more general classification of CTDG-focused methods, comparing with DTDG-focused ones based on approaches used for learning evolution patterns.
Temporal random-walks models. The approach implies including the time dependency directly in a sequence of nodes generated by random walks. Such methods create a corpus of walks over time (so-called "temporal random walks") with respect to the order of nodes/edges appearance in the graph. Based on this idea, authors of [50] leverage a custom attention mechanism to learn the importance between a node and its temporal random-walk-based neighbors. Using this work, [17] proposes several methods to select the subsequent nodes connected to a starting one. A promising method for link prediction task was proposed by the authors of [61] who have developed Causal Anonymous Walks (CAWs), constructed from temporal random walks. CAWs adopt a novel anonymization strategy that replaces node identities with the hitting counts of the nodes based on a set of sampled walks to keep the method inductive.
The model outperforms previous methods in both inductive and transductive settings.
Local neighborhood models. When interactions happen (node or edge adding and removal), models in this class update embeddings of the relevant nodes by aggregating information from their new neighborhoods. In [33,64], authors utilize a GCN-based aggregation scheme and propagate changes to higher-order neighbors of interacting nodes. To cope with information asymmetry, the authors of [16] propose to determine the priority of the nodes that receive the latest interaction information. Models from [10,51] embed dynamic network for recommendation systems in similar ways. Several recent works [5,45,58] consider interactions between nodes as stochastic processes with probabilities depending on the network statistics.
Memory-based models. The core idea of this class of temporal network embeddings lies in extending existing models by using special memory modules storing a history of interactions for each node. Methods vary from updating LSTM [59] to using augmented matrices of interactions [66]. APAN (Asynchronous Propagation Attention Network) [56] aims to store detailed information about k-hop neighborhood interactions of each node in so-called "mailboxes". The recently developed temporal graph network [12] proposes a flexible framework, which consists of several independent modules and generalizes other recent CTDGs-focused models such as [10,45,51]. Combining the advantages of JODIE [51] and temporal graph attention (TGAT) [10], TGN introduces the node-wise memory into the temporal aggregate phase of TGAT, showing stateof-the-art for industrial tasks results.
Because of the potential of CTDG-based models [22] and necessity to apply the model to transaction data, we focus on developing a model in this class, while keeping in mind efficient DTDG models [62]. In what follows, we describe the idea of improving existing state-of-the-art CTDG models by properly fusing memory-based, neighborhood and interaction information under unified framework, thus combining multiple best practices of CTDG methods.

PROPOSED APPROACH
As stated above in our literature survey, the best-known model for temporal graph prediction tasks, namely, TGN [12], relies primarily on propagating information ("message") through edges and generates node embeddings from occurring edges in a straightforward manner. At the same time, it was noticed that the CAW [61] shows excellent performance in link prediction tasks by learning edge representations. Unfortunately, the latter model cannot be directly applied to the extraction of node embeddings, and, consecutively, downstream graph machine learning tasks, such as node classification.
Hence, in this paper, we propose a novel network embedding taking the best out of the TGN framework and the CAW edge encodings (Fig. 2) to improve the quality of edge and node classification. Our contribution is to leverage the highly informative edge embedding generated in CAW. This will allow us to refine various functions of the TGN framework, including generating messages and embeddings, as well as aggregating memory.

Model details
Below, we describe our model by listing and describing all modules in a consecutive fashion. In the following descriptions and equations, LSTM refers to the Long Short-Term Memory type RNN, GRU is the Gated Recurrent Unit, "attn" and "self-attn" is attention and self-attention, respectively. A walk on a graph, performed backwards in time on a temporal graph (time-inverse walk) with length is denoted by where ∈ V is the node entered at step , is the time of step , ( , ) is the -th node-time pair. In future, we will refer to the elements of the -th step as (1) = and (2) = .
The -timed -hop neighborhood of -th node is a set of nodes which can be reached from in steps if one walks on the edges existing prior to time , without regard for their direction: (2) 4.1.1 Neighborhood Edge Feature (NEF) Generator). This trainable module and its integration in the following modules are intended to improve the predictive capabilities of the original TGN framework, which is our core contribution. It generates highly informative feature representations of pairs of nodes for any given moment in time. For a pair of nodes and , the output of this module as ( ) is computed as follows. At first, we sample an equal number > 1 of the time-inverse walks for both nodes in order to capture information about the edge neighborhoods. All sampled walks have identical lengths (1-2 steps typically). A constant decay hyperparameter, which regulates how strongly the sampling process prioritizes more recent connections, is used during sampling of all walks.
The walk sets and are generated for edge between nodes and . consists of sampled time-inverse walks , ∈ {1, ..., }. Each edge for each walk is sampled with the probabilities proportional to ( △ ), where is the decay parameter, and △ is the time difference between edges. The next step makes the sampled walks anonymous in order to limit extracted information to the edge neighborhood alone. Each node from is replaced by a pair of vectors that encode a positional frequency of the node in each corresponding position of walks in and . The encoding vector of positional frequencies relative to the walks of for node is defined as: , ∈ {1, ..., }}|}. ( This equation simply specifies that the node index is encoded as a vector, so that each its -th component is equal to the number of times where is the -th node of some walk in .
The anonymization of walks [61] is achieved by using the defined function to transform node indices in walks. Each position in any walk of containing is replaced by Similarly, any value in walks of filled by is replaced with ( ). In the remaining part of this section we write = , assuming the known orientation of the edge.
The remaining steps simply attempt to transform the "anonymous walk" representation of the node pair to a more compressed and usable state. Each obtained representation of each position of each walk, i.e. pairs of { ( , ), ( , )}, are fed through separate instances of the same two-layered multi-layered perceptron (MLP) and sum-pooled: Then, this representation is concatenated with the time-difference encoding and node or edge features of the corresponding step: where being -th position of walk . The encoded walks are aggregated across all walks for the node pair : where | ∪ | is amount of walks in a set ∪ , and agg is either self-attention module or identity for mean pooling aggregation.

Message store.
In order to apply the gradient descent, a memory of a node should be updated after it is passed as a training instance. Let's say an event associated with the -th node has occurred, e.g. an edge involving has been passed for training in the current batch. Then, all information about the batch transactions involving node will be recorded in the message store after the batch inference, replacing the existing records for . This information is used and updated during the processing of the next batch involving .

Message Generator.
When the model processes a batch containing a node, all transaction information about edges associated with this node is pulled. For each transaction between and some other node at a time , a message for a node is computed as a concatenation of the current memory vectors of the nodes, edge features, time-related features, and neighborhood edge features of the corresponding edge: where is the index of a transaction between nodes and , ( − ) is a memory state of node at time − of last memory update for node , and ( ) is a neighborhood edge feature vector for edge . The usage of NEF features here is our novel idea aimed to include more information about the type of the update message. The generated messages will update memory states of the batch nodes. Note that, aside from the basic and non-trainable concatenation, other choices for the message function, like MLPs are possible.

Message Aggregator.
To perform a memory update on a node at time , its message representation is obtained by aggregating all currently stored messages timestamps 1 < 2 < ... < which are related to this node: Here aggregation function can be computed as a mean of generated messages. It is still possible to use the most recent message ( ) as a value of ( ).

Memory
Updater. The message generator and aggregator let the model to encode useful transaction information as memory vectors. In particular, the memory state vector for node is updated at time by applying an RNN-type model to the concatenation of the received message and previous memory state: where RNN is either GRU or LSTM, and initial state (0) is initialized with a random vector.

Embedding
Generator. This module generates the node embedding based on memory states of the node and its k-hop neighborhood, features of the neighborhood, and NEF representations of "virtual" edges between the node and its direct neighbors. Resulting node embeddings can be viewed as autonomous node representations for classification. We propose, similarly to message generator, to add NEF features to node in order to let the model better discriminate neighbors on their relevance or type: where ( ) is a feature vector of node at time , and ℎ 0 and ℎ 1 are the units of the neural network. It is typical to use MLPs as ℎ 0 and ℎ 1 , but the following concatenation by self-attention is more preferable to capture complex dependencies involving NEFs.

Embedding
Decoder. This is the final module, which transforms node embeddings into prediction results for downstream tasks. In this paper, we always use MLP with 3 layers with only node embeddings as inputs, and sigmoid or softmax output layers. For example, the multi-class classification problem for node at time is solved using the following equation: Similarly, the edge prediction task is solved as follows: ( ) = Sigmoid(MLP(concat( ( ) ( )))).

Core idea and novelty of proposed model
Let us provide a high-level description of the model (Fig. 2). It learns to generate a temporal embedding for each node and decode embeddings into inputs for each classification task. The model assigns a memory vector to each node and generates each node embedding by aggregating memory vectors and other relevant features in a neighborhood of the node. Node memory vectors describe relevant information about interactions involving the node. They are updated using specified node messages, which, in turn, encode information about the last transaction involving the node. This design allows employing gradient descent for training memory and message generating modules. A decoder MLP transforms a pair of node embeddings into the probability of a temporal edge existing between the nodes. Similarly, it can be trained to transform a single node embedding into probabilities of the node belonging to each of the existing classes for a node classification problem. There are two main modifications of the proposed model compared to the baseline TGN framework [12]. First, we change the message generating function, which provides the model with an additional way to differentiate the messages based on their relevance. The NEF features of an edge contain information about the walk correlation of the two nodes. As it can be used for very accurate link prediction [61], it may be also used to classify some messages as being irrelevant, and diminish their effect on the memory update.
Second, while the original version of the embedding module [12] allows treating different k-hop neighbors of the node differently if using attention for aggregation, it might be beneficial to provide the NEF features of connections between the node and its closest neighbors. As a result, we again take into account the walk correlation between the node and its neighbor, so that the differences in neighborhood type can be more evident.

EVALUATION FRAMEWORK
In this section, we discuss our research methodology. This involves the evaluation framework, evaluation pipeline training settings, hyper-parameter choice and description of temporal networks used for the evaluation.

Pipeline
The main contribution of the proposed framework (Fig. 1) is an easy-to-use unified data processing toolkit for accurate evaluation of temporal network embeddings in downstream tasks under common training settings, which allows to remove contradictions in experimental results reported in research articles in the field. The source code of this tool is publicly available 1 . It is focused on transforming any graph into a universal format that is afterwards fed into the pipeline in either DTDG or CTDG format with following features: • selection of precise batching options; • preparing data for both inductive and transductive settings (with support for bipartite data); • interfacing with any kind of graph embedding models using an interface for model-framework communication, treating network embedding models as a black-box, which is then passed on to such basic methods as train_model, pre-dict_edge_probabilities, etc.

Training settings
5.2.1 Negative sampling. It is common to design include negative edge sampling into evaluation of temporal graph prediction tasks. It is employed for balancing edges and non-connected pairs of nodes. We follow a standard setting, where the number of negative samples is equal to the number of edges.

Batch specification.
The model is trained by passing edges (which includes negative samples) in batches sorted in chronological order.

5.2.3
Training with temporal data masking. We use 80%-10%-10% train-val-test split with randomized training sets, supported via several data masking schemes. Node masking hides nodes (as well as all connected edges) from training data. The masked nodes for transductive tasks are also removed from the validation and test data, while for inductive tasks masked nodes remain in validation and test data. Edge masking removes a fixed percentage of random edges from the whole dataset. We use two options for both masking techniques: the percentage of masked information is 10% and 70%, representing dense and sparse training settings, respectively. In particular, we report the results for three combinations, in which at least one mask is large, namely 10%-75% node-edge masking, 75%-10% and 75%-75%.

Runs and validation.
Each model/setting/dataset combination was run 10 times with random seeds and node/edge masking.

Comparison with the state-of-the-art models
In order to evaluate our model (Fig. 2), we compare it to the baseline TGN framework architectures (configured as described in [12]), DyRep (supporting long-term and short-term time scales of graph evolution and scaling to large graphs) [45] and Jodie (working with bipartite graphs using node embeddings for predictive tasks with future time lag) [51]. Though these models can construct high-quality historic embeddings using information propagated from nodes, they do not take into account edge information and interaction patterns. Hence, we could demonstrate the advantages of our approach in identical training settings in comparison to other state-of-the-art temporal network embeddings. It is important to emphasize that the CAW model cannot be used here because it was

Datasets
In our study, we focus on well-known benchmark datasets for temporal networks. Their descriptive statistics are presented in Table 1, taking into account specifics of computing statistics for bipartite graphs of user-item interactions [37].
Labeled datasets. Both Reddit [32] and Wikipedia [15] datasets are bipartite graphs representing interactions between users (source) and web resources (target), like posting to a subreddit or editing a wiki page. Text content is encoded as edge features to provide context. Both datasets have a binary target, indicating whether a user was banned or not.
Non-labeled datasets. The UCI [40] dataset contains a communication history for the students' forum. The Enron [23] dataset is constructed over internal e-mail communication of Enron company employees. The Ethereum dataset 2 contains a directed graph of Ethereum blockchain transactions, which were collected from larger public BigQuery dataset 3 ; we collected a dataset containing all transactions, which occurred during a 9-hour span. All of the non-labeled datasets are non-bipartite, with no additional features for edges or nodes.

ABLATION STUDY
In this Section an ablation study of the proposed architecture (Fig. 2) is provided to explore the impact of each submodule. Below we specify several additional modules added to interaction features processing pipeline for temporal network embedding via NEF generator,which outlined in Subsection 4.1.1: (Msg) NEF-Message concatenating NEF features and messages; (Emb) NEF-Embed generates embeddings with NEF features; (RNN) NEF-LSTM including Bi-LSTM into NEF generator, instead of mean pooling across walk elements.

EXPERIMENTAL RESULTS
In this section, we compute mean AUC-ROC and AP for our model and compare it to three other state-of-the-art models (DyRep, Jodie, TGN). Tables 3-Table 6 show results for edge prediction task in transductive and inductive settings respectively. Best results are given in bold, while second best are underlined. "Node mask" and "Edge mask" columns specify the portion of nodes or edges used for model testing.    As evident from above figures, the proposed approach (Fig. 2) consistently outperforms almost all other models. For example, our model achieves the best results in transductive edge prediction (Table 3) except one case, where DyRep achieved better results. Moreover, our method achieves the best results on all datasets for inductive edge prediction (Table 5). It seems that capturing networkwide common patterns with CAW-based features generally offers an improvement over existing models, although in some cases information added by these features may be duplicated by other encoding modules.   Table 7 and Table 8 present the results for node classification tasks in transductive and inductive settings respectively. Though our model generally outperforms existing methods, there is one exception. Indeed, Jodie [51] shows the best node classification quality on Wikipedia dataset. As a matter of fact, the Jodie model incorporates both static and dynamic node embeddings, which might contributed to better encoding of interests and behavior patterns of the Wikipedia editors

BANKING RESULTS
As an industrial application of the proposed framework, we chose a pre-existing problem posited by a major European banks with a corresponding dataset which consists of transactions between different companies. Each transaction has a timestamp so we can treat it as a temporal edge. This allows us to consider only "edge events". We use the following edge features: transaction amount, timestamp and type of transaction, (e.g. credit or state duty). We one-hot encode the type of transaction, which gives us 50 edge features in total: 49 from one-hot encoding and 1 from transaction amount. These features were normalized.
At first, we compared our model with the original TGN [12] on a link prediction task. We obtain the training and validation datasets consisting of transactions from a regional subdivision of a major European bank for a time period about one week. This dataset contain about 2 millions of transactions. We select about 40000 newest transactions to test dataset and all the rest is given to the train i. e. test dataset transactions are always newer then train transactions. We use the following procedure to check performance of our model on test dataset. First, we split training data into small chunks (400 batches) and gradually increase the number of chunks  Fig. 3.
We can see very high performance of dynamic graph approach models. It can be noticed that performance of our model grows a little bit slower than for TGN [12], but after about 600000 transactions our model takes the lead. This slight time lag in training is due to our model taking additional time to learn extra information about interactions between nodes via learning CAW [61] part of our model. This extra information allows to gain better results in the end of training. Table 9: Node classification results based on node embeddings obtained from TGN and our proposed model with feature dimension being multiple of 8 × 2 shape Node Embeddings AUC-ROC TGN 0.626 ± 0.072 Our, = 10 0.653 ± 0.049 Our, = 20 0.598 ± 0.067 Our, = 100 0.578 ± 0.094 The architecture of our model and TGN [12] model allows us to produce node embeddings which can the be used as input for a wide range of possible downstream tasks. For our experiment we have taken prediction of company default as the downstream task. The dataset contains about 5000 companies, some of which will go bankrupt in a specific period of time in the future (180 days) and others will not. To train a classification model we used LightGBM [29]. The obtained results were averaged over with 5-fold cross-validation and presented in Table 9.
Here our best model has 2.7% higher AUC ROC when compared to the TGN [12]. That means that extra information about interactions between nodes captured by CAW [61] part of our model is useful for classification problems. We see also that best performance is observed with small CAW messages shape. This is related to the structure of our dataset. It's number of edges is comparable with number of nodes, thus the graph is sparse. Bigger CAW messages shapes can be useful for more dense graphs.

CONCLUSION
In this work, we proposed a novel model for temporal graph embedding (Fig. 2) that shows improvement over existing methods [12,45,51] on various prediction tasks while preserving the ability to generate node embeddings. Moreover, we implemented a novel experimental framework (Fig. 1) that can process most kinds of graph data and an arbitrary dynamic graph inference model. Experimental study demonstrates the applicability of our method to solving various node/edge prediction tasks on temporal networks, and significantly improving the existing results.
In future, it is possible to improve the performance of our framework for its application to the real-life temporal graph of bank's transactions with tens of billions of nodes. It is necessary to study efficient node/edge sampling strategies, choosing those that overcome the limitations of current models when scaling to large graphs while preserving highly-informative edge features propagation.