{'fraction_non_alphanumeric': 0.027140516181892668, 'fraction_numerical': 0.0036870135190495697, 'mean_word_length': 5.145374449339207, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep attributed network embedding by mutual information maximization

There is a growing body of literature that recognizes the importance of network embedding. It intends to encode the graph structure information into a low-dimensional vector for each node in the graph, which benefits the downstream tasks. Most of recent works focus on supervised learning. But they are usually not feasible in real-world datasets owing to the high cost to obtain labels. To address this issue, we design a new unsupervised attributed network embedding method, deep attributed network embedding by mutual information maximization (DMIM). Our method focuses on maximizing mutual information between the hidden representations of the global topological structure and the node attributes, which allows us to obtain the node embedding without manual labeling. To illustrate the effectiveness of our method, we carry out the node classification task using the learned node embeddings. Compared with the state-of-the-art unsupervised methods, our method achieves superior results on various datasets.


Introduction
Network embedding is important for a wide range of scientific and industrial processes. While maintaining the metric proximity among nodes, for each node, it aims to learn a low-dimensional vector. The learned representations can be applied as latent features for following various tasks, including node classification [1], graph classification and link prediction [2].
Early works for unsupervised network embedding rely on walk-based objective [2,3,4,5], sometimes further simplified to reconstruct adjacency information [1,6] or node attributes [7]. They intend to make node representations are close if nodes are close in the input graph. However, these works only utilize the topological structure while discarding node's rich attributes. The node attributes contain rich information, which can indicate the different relationships between nodes. Currently, the dominant algorithms founded on graph neural networks(GNN) [1] utilize more expressive graph convolutional encoders compared with conventional methods. One of the representative methods is GraphSAGE [5], which combines graph convolutional encoder and DeepWalk-like objectives. Recent work DGI [8] first applies contrastive learning on the graph, which magnifies mutual information between local node representations and global graph representations. However, GNN based methods rely on aggregated-based scheme, which can't employ deep network layers and apply to large-scale network. Due to its over-smoothing problem, too deep network architecture cannot be used. To overcame above challenges, we design a new unsupervised learning method, deep attributed network embedding by mutual information maximization (DMIM). The mutual information maximization between two types of hidden representations of graph topological structure and of node's rich attributes can encourage the model to better fuse these two types information. The following is a summary of the major contributions:  We propose an unsupervised network embedding method to capture the network topological structure and node attributes of arbitrary size network;  The proposed method can employ more deep network layers that GNN based methods.  We reveal that the node embeddings derived from the proposed model can obtain superior node classification results compared with other unsupervised methods on various datasets;

Materials and Methods
Motivated by the core idea of contrastive learning, we design a new deep unsupervised attributed network embedding method, DMIM, to pick up topological structure information and node's rich attributes, and dig up the node representations. Our method is on the basis of the maximization of mutual information between the hidden representations of node attributes and of the topological structure from the same node. The proposed method first utilizes random walks to extract the global topological structure, then obtains two types of hidden representation from the encoder, at last fusion these two types of hidden representations on the basis of mutual information maximization.

Problem definition
Given G V, A, X indicated an attributed network, and in the network the number of nodes is N |V|. A ∈ ℛ is the adjacency matrix with size N * N, is the attributed matrix. In detail, X is the attributes of the i-th node, which is the i-th row of X . Attributed network embedding intends to learn the low-dimensional embedding, as denoted by Z , such that Z can preserve the network structure and node's rich features.

Global topological structure
In that the adjacency matrix can only represent the local topological structure, we extract the global topological structure S ∈ ℛ founded on random walks. In detail, we first use random walks to collect enough node sequences. Then we extract the global context of the node from these node sequences. Specifically, if two nodes are similar, they are more likely to appear in the context of each other. Due to the above operations, we get the global topological structure of each node.

Autoencoder
For capturing the highly non-linear structure, we use autoencoder to restore the global structure and node attributes respectively. h , ℎ 1 Here x ∈ ℛ is the attribute of the i-th node, h ∈ ℛ is the hidden representation of the attribute of the i-th node from the encoder, and ∈ is the reconstructed attribute of the i-th node from the decoder. σ . denotes the non-linear activation function. we get the loss function ℒ for attribute reconstruction using matrix form. Here f ∈ R is the hidden representation of the global topological structure of the i-th node from the encoder.
Additionally, we reconstruct the adjacency matrix to ensure locality. Specifically, if two nodes are close, their hidden representations are also close, and vice versa.
is hidden representations of the node attributes, and F ∈ ℛ is the hidden representations of the global topological structure.

Fusion strategy
For each node, we have two types of hidden representations, which come from the topological structure and node attributes respectively. These hidden representations don't contain information from each other. We fusion these two types of hidden representations on the basis of mutual information maximization. Specifically, a discriminator is introduced to help teach the encoders, which helps our model to yield the satisfied node representations. The discriminator teachs the encoders to magnify the mutual information between the hidden representations of the topological structure and of the node's rich attributes for each node. Here a discriminator ∶ ℛ ℛ → ℛ , is applied to represent the probability scores of the topology-attribute pair. By combining the hidden representation of node attribute with another hidden representation from other nodes, we draw the negative samples.

Object function
Therefore, to get the network embedding, we optimize the following objection function jointly: ℒ α ℒ ℒ ℒ ℒ 7 Here, α, β, γ is the hyperparameter. By minimizing this objection function, we can obtain h and f , then we concatenate them as the final low-dimensional representation of the node.

Results & Discussion
We employ the learned node embeddings on node classification tasks to evaluate our method. The classification outcomes are compared with other methods. We use a linear evaluation of the learned node embeddings, as is usual practice [1,5]. On three well-known datasets, we employ these node embeddings to optimize a logistic regression model to handle multiclass node classification tasks.

Dataset & baselines
Cora, Citeseer and Pubmed are three typical citation network benchmark datasets that we use. In all of these datasets, nodes are bag of words representations and edges are citation relations. A class label is assigned to each node. For comparison, we use Label Propagation(LP) [9] and DeepWalk [3], two stateof-the-art unsupervised approaches. We also present the results of training logistic regression using solely natural input features, as well as the concatenation of DeepWalk embeddings and the natural features of node. In addition to unsupervised methods, we also use strong supervised methods for comparison: Planetoid [10] and FastGCN [11]. Table 1 shows the node classification results on three benchmark datasets. After 20 runs of training, we provide the mean classification accuracy on the test dataset of our approach, and we reuse the metrics from Kipf & Welling [1] for the performance of DeepWalk, as well as Label Propagation (LP) [9] and Planetoid [10]. We also reuse the metrics already in Jiao & Xiong [12] for the performance of FastGCN  [11]. For all the there datasets, we set output dimension to 200 for our method. As per the outcomes, our model outperforms DeepWalk in node classification and is comparable to supervised learning methods.

Conclusions
In this paper, we propose a novel unsupervised learning method, deep attributed network embedding by mutual information maximization (DMIM), to learn node embeddings. The mutual information maximization between two types of hidden representations of graph topological structure and of node attributes can encourage the model to better fuse these two types information. This unsupervised learning model can learn node embeddings that are task-independent. In addition, it can employ deep network layers and apply to large-scale datasets than GNN based methods. We perform node classification tasks with the learned node embeddings to illustrate the model's effectiveness. The results show that our unsupervised model outperforms DeepWalk in classification and achieves comparable results with supervised learning approaches on serval tested datasets.