{'fraction_non_alphanumeric': 0.0328663531921082, 'fraction_numerical': 0.007804546996946046, 'mean_word_length': 4.7753359462486005, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Heterogeneous Graph Attention Network

Graph neural network, as a powerful graph representation technique based on deep learning, has shown superior performance and attracted considerable research interest. However, it has not been fully considered in graph neural network for heterogeneous graph which contains different types of nodes and links. The heterogeneity and rich semantic information bring great challenges for designing a graph neural network for heterogeneous graph. Recently, one of the most exciting advancements in deep learning is the attention mechanism, whose great potential has been well demonstrated in various areas. In this paper, we first propose a novel heterogeneous graph neural network based on the hierarchical attention, including node-level and semantic-level attentions. Specifically, the node-level attention aims to learn the importance between a node and its metapath based neighbors, while the semantic-level attention is able to learn the importance of different meta-paths. With the learned importance from both node-level and semantic-level attention, the importance of node and meta-path can be fully considered. Then the proposed model can generate node embedding by aggregating features from meta-path based neighbors in a hierarchical manner. Extensive experimental results on three real-world heterogeneous graphs not only show the superior performance of our proposed model over the state-of-the-arts, but also demonstrate its potentially good interpretability for graph analysis.

web. Graph neural network (GNN), as a powerful deep representation learning method for such graph data, has shown superior performance on network analysis and aroused considerable research interest. For example, [10,20,24] leverage deep neural network to learn node representations based on node features and the graph structure. Some works [6,14,18] propose the graph convolutional networks by generalizing the convolutional operation to graph. A recent research trend in deep learning is the attention mechanism, which deals with variable sized data and encourages the model to focus on the most salient parts of data. It has demonstrated the effectiveness in deep neural network framework and is widely applied to various applications, such as text analysis [1], knowledge graph [25] and image processing [38]. Graph Attention Network (GAT) [35], a novel convolution-style graph neural network, leverages attention mechanism for the homogeneous graph which includes only one type of nodes or links.
Despite the success of attention mechanism in deep learning, it has not been considered in the graph neural network framework for heterogeneous graph. As a matter of fact, the real-world graph usually comes with multi-types of nodes and edges, also widely known as heterogeneous information network (HIN) [28]. For convenience, we uniformly call it heterogeneous graph in this paper. Because the heterogeneous graph contains more comprehensive information and rich semantics, it has been widely used in many data mining tasks. Meta-path [32], a composite relation connecting two objects, is a widely used structure to capture the semantics. Taking the movie data IMDB 1 shown in Figure 1(a) as an example, it contains three types of nodes include movie, actor and director. A relation between two movies can be revealed by meta-path Movie-Actor-Movie (MAM) which describes the co-actor relation, while Movie-Director-Movie (MDM) means that they are directed by the same director. As can be seen, depending on the meta-paths, the relation between nodes in the heterogeneous graph can have different semantics. Due to the complexity of heterogeneous graph, traditional graph neural network cannot be directly applied to heterogeneous graph.
Based on the above analysis, when designing graph neural network architecture with attention mechanism for heterogeneous graph, we need to address the following new requirements.
Heterogeneity of graph. The heterogeneity is an intrinsic property of heterogeneous graph, i.e., various types of nodes and edges. For example, different types of nodes have different traits and their features may fall in different feature space. Still taking IMDB as an example, the feature of an actor may involve in sex, age and nationality. On the other hand, the feature of movie may involve in plot and actors. How to handle such complex structural information and preserve the diverse feature information simultaneously is an urgent problem that needs to be solved.
Semantic-level attention. Different meaningful and complex semantic information are involved in heterogeneous graph, which are usually reflected by meta-paths [32]. Different meta-paths in heterogeneous graph may extract diverse semantic information. How to select the most meaningful meta-paths and fuse the semantic information for the specific task is an open problem [4,19,26]. Semantic-level attention aims to learn the importance of each metapath and assign proper weights to them. Still taking IMDB as an example, The Terminator can either connect to The Terminator 2 via Movie-Actor-Movie (both starred by Schwarzenegger) or connect to Birdy via Movie-Year-Movie (both shot in 1984). However, when identifying the genre of the movie The Terminator, MAM usually plays more important role, rather than MYM. Therefore, treating different meta-paths equally is unpractical and will weaken the semantic information provided by some useful meta-paths.
Node-level attention. In a heterogeneous graph, nodes can be connected via various types of relation, e.g., meta-path. Given a meta-path, each node has lots of meta-path based neighbors. How to distinguish the subtle difference of there neighbors and select some informative neighors is required. For each node, node-level attention aims to learn the importance of meta-path based neighbors and assign different attention values to them. Still taking IMDB as an example, when using the meta-path Movie-Director-Moive (the movies are with the same director), The Terminator will connect to Titanic and The Terminator 2 via director James Cameron. To better identify the genre of The Terminator as sci-fi movie, the model should pay more attention to The Terminator 2, rather than Titanic. Therefore, how to design a model which can discover the subtle differences of neighbors and learn their weights properly will be desired.
In this paper, we propose a novel Heterogeneous graph Attention Network, named HAN, which considers both of node-level and semantic-level attentions. In particular, given the node features as input, we use the type-specific transformation matrix to project different types of node features into the same space. Then the node-level attention is able to learn the attention values between the nodes and their meta-path based neighbors, while the semantic-level attention aims to learn the attention values of different meta-paths for the specific task in the heterogeneous graph. Based on the learned attention values in terms of the two levels, our model can get the optimal combination of neighbors and multiple meta-paths in a hierarchical manner, which enables the learned node embeddings to better capture the complex structure and rich semantic information in a heterogeneous graph. After that, the overall model can be optimized via backpropagation in an end-to-end manner.
The contributions of our work are summarized as follows: • To our best knowledge, this is the first attempt to study the heterogeneous graph neural network based on attention mechanism. Our work enables the graph neural network to be directly applied to the heterogeneous graph, and further facilitates the heterogeneous graph based applications.
• We propose a novel heterogeneous graph attention network (HAN) which includes both of the node-level and semantic-level attentions. Benefitting from such hierarchical attentions, the proposed HAN can take the importance of nodes and meta-paths into consideration simultaneously. Moreover, our model is high efficiency, with the linear complexity with respect to the number of meta-path based node pairs, which can be applied to large-scale heterogeneous graph.
• We conduct extensive experiments to evaluate the performance of the proposed model. The results show the superiority of the proposed model by comparing with the state-of-the-art models. More importantly, by analysing the hierarchical attention mechanism, the proposed HAN demonstrates its potentially good interpretability for heterogeneous graph analysis.

RELATED WORK 2.1 Graph Neural Network
Graph neural networks (GNNs) which aim to extend the deep neural network to deal with arbitrary graph-structured data are introduced in [10,24]. Yujia Li et al. [20] proposes a propagation model which can incorporate gated recurrent units to propagate information across all nodes. Recently, there is a surge of generalizing convolutional operation on the graph-structured data. The graph convolutional neural work generally falls into two categories, namely spectral domain and non-spectral domain. On one hand, spectral approaches work with a spectral representation of the graphs. Joan Bruna et al. [2] extends convolution to general graphs by finding the corresponding Fourier basis. Michaël et al. [6] utilizes K-order Chebyshev polynomials to approximate smooth filters in the spectral domain. Kipf et al. [18] proposes a spectral approach, named Graph Convolutional Network, which designs a graph convolutional network via a localized first-order approximation of spectral graph convolutions. On the other hand, we also have non-spectral approaches, which define convolutions directly on the graph, operating on groups of spatially close neighbors. Hamilton et al. [14] introduces GraphSAGE which performs a neural network based aggregator over a fixed size node neighbor. It can learn a function that generates embeddings by aggregating features from a nodeâȂŹs local neighborhood.
Attention mechanisms, e.g., self-attention [34] and soft-attention [1], have become one of the most influential mechanisms in deep learning. Some previous works introduce the attention mechanism for graph based applications, e.g., the recommendation [15,16]. Inspired by attention mechanism, Graph Attention Network [35] is proposed to learn the importance between nodes and its neighbors and fuse the neighbors to perform node classification. However, the above graph neural network cannot deal with various types of nodes and edges and can only be applied to the homogeneous graphs.

Network Embedding
Network embedding, i.e., network representation learning (NRL), is proposed to embed network into a low dimensional space while preserving the network structure and property so that the learned embeddings can be applied to the downstream network tasks. For example, the random walk based methods [12,23], the deep neural network based methods [36], the matrix factorization based methods [22,37], and others, e.g., LINE [33]. However, all these algorithms are proposed for the homogeneous graphs. Some elaborate reviews can be found in [5,11].
Heterogeneous graph embedding mainly focuses on preserving the meta-path based structural information. ESim [26] accepts userdefined meta-paths as guidance to learn vertex vectors in a userpreferred embedding space for similarity search. Even through ESim can utilize multiple meta-paths, it cannot learn the importance of meta-paths. To achieve the best performance, ESim needs to conduct grid search to find the optimal weights of hmeta-paths. It is pretty hard to find the optimal combination for specific task. Meta-path2vec [7] designs a meta-path based random walk and utilizes skip-gram to perform heterogeneous graph embedding. However, metapath2vec can only utilize one meta-path and may ignore some useful information. Similar to metapath2vec, HERec [27] proposes a type constraint strategy to filter the node sequence and capture the complex semantics reflected in heterogeneous graph. HIN2Vec [9] carries out multiple prediction training tasks which learn the latent vectors of nodes and meta-paths simultaneously. Chen et al. [3] proposes a projected metric embedding model, named PME, which can preserve node proximities via Euclidian Distance. PME projects different types of node into the same relation space and conducts heterogeneous link prediction. To study the problem of comprehensive describe heterogeneous graph, Chen et al. [29] proposes HEER which can embed heterogeneous graph via edge representations. Fan et al. [8] proposes a embedding model metagraph2vec, where both the structures and semantics are maximally preserved for malware detection. Sun et al. [30] proposes meta-graph-based network embedding models, which simultaneously considers the hidden relations of all meta information of a meta-graph. In summary, all these aforementioned algorithms do not consider the attention mechanism in heterogeneous graph representation learning.

PRELIMINARY
A heterogeneous graph is a special kind of information network, which contains either multiple types of objects or multiple types of links.
Definition 3.1. Heterogeneous Graph [31]. A heterogeneous graph, denoted as G = (V, E), consists of an object set V and a link set E. A heterogeneous graph is also associated with a node type mapping function ϕ : V → A and a link type mapping function ψ : E → R. A and R denote the sets of predefined object types and link types, where |A| + |R| > 2.
Example. As shown in Figure 1(a), we construct a heterogeneous graph to model the IMDB. It consists of multiple types of objects ( Actor (A), Movie (M), Director (D)) and relations (shoot relation between movies and directors, role-play relation between actors and movies).
In heterogeneous graph, two objects can be connected via different semantic paths, which are called meta-paths.
Example. As shown in Figure 1(a), two movies can be connected via multiple meta-paths, e.g., Movie-Actor-Movie (MAM) and Movie-Director-Movie (MDM). Different meta-paths always reveal different semantics. For example, the MAM means the coactor relation, while Movie-Director-Movie (MDM) means they are directed by the same director.
Given a meta-path Φ, there exists a set of meta-path based neighbors of each node which can reveal diverse structure information and rich semantics in a heterogeneous graph. Definition 3.3. Meta-path based Neighbors. Givien a node i and a meta-path Φ in a heterogeneous graph, the meta-path based neighbors N Φ i of node i are defined as the set of nodes which connect with node i via meta-path Φ. Note that the node's neighbors includes itself.
Example. Taking Figure 1(d) as an example, given the meta-path Movie-Actor-Movie, the meta-path based neighbors of m 1 includes m1 (itself), m 2 and m 3 . Similarly, the neighbors of m 1 based on meta-path Movie-Director-Movie includes m 1 and m 2 . Obviously, meta-path based neighbors can exploit different aspects of structure information in heterogeneous graph. We can get meta-path based neighbors by the multiplication of a sequences of adjacency matrices.
Graph neural network has been proposed to deal with arbitrary graph-structured data. However, all of them are designed for homogeneous network [18,35]. Since meta-path and meta-path based neighbors are two fundamental structures in a heterogeneous graph, next, we will present a novel graph neural network for heterogeneous graph data, which is able to exploit the subtle difference of nodes and meta-paths. The notations we will use throughout the article are summarized in Table 1.

THE PROPOSED MODEL
In this section, we propose a novel semi-supervised graph neural network for heterogeneous graph. Our model follows a hierarchical attention structure: node-level attention → semantic-level attention. Figure 2 presents the whole framework of HAN. First, we propose a node-level attention to learn the weight of meta-path based neighbors and aggregate them to get the semantic-specific node embedding.

Notation
Explanation The final embedding After that, HAN can tell the difference of meta-paths via semanticlevel attention and get the optimal weighted combination of the semantic-specific node embedding for the specific task.

Node-level Attention
Before aggregating the information from meta-path neighbors for each node, we should notice that the meta-path based neighbors of each node play a different role and show different importance in learning node embedding for the specific task. Here we introduce node-level attention can learn the importance of meta-path based neighbors for each node in a heterogeneous graph and aggregate the representation of these meaningful neighbors to form a node embedding.
Due to the heterogeneity of nodes, different types of nodes have different feature spaces. Therefore, for each type of nodes (e.g.,node with type ϕ i ), we design the type-specific transformation matrix M ϕ i to project the features of different types of nodes into the same feature space. Unlike [13], the type-specific transformation matrix is based on node-type rather than edge-type. The projection process can be shown as follows: where h i and h ′ i are the original and projected feature of node i, respectively. By type-specific projection operation, the node-level attention can handle arbitrary types of nodes.
After that, we leverage self-attention [34] to learn the weight among various kinds of nodes. Given a node pair (i, j) which are connected via meta-path Φ, the node-level attention e Φ i j can learn the importance e Φ i j which means how important node j will be for node i. The importance of meta-path based node pair (i, j) can be formulated as follows: Here att node denotes the deep neural network which performs the node-level attention. Given meta-path Φ, att node is shared for all meta-path based node pairs. It is because there are some similar connection patterns under one meta-path. The above Eq. (2) shows that given meta-path Φ, the weight of meta-path based node pair (i, j) depends on their features. Please note that, e Φ i j is asymmetric, i.e., the importance of node i to node j and the importance of node j to node i can be quite difference. It shows node-level attention can preserve the asymmetry which is a critical property of heterogenous graph.
Then we inject the structural information into the model via masked attention which means we only calculate the e Φ i j for nodes j ∈ N Φ i , where N Φ i denotes the meta-path based neighbors of node i (include itself). After obtaining the importance between meta-path based node pairs, we normalize them to get the weight coefficient α Φ i j via softmax function: where σ denotes the activation function, ∥ denotes the concatenate operation and a Φ is the node-level attention vector for meta-path Φ.
As we can see from Eq. (3), the weight coefficient of (i, j) depends on their features. Also please note that the weight coefficient α Φ i j is asymmetric which means they make different contribution to each other. Not only because the concatenate order in the numerator, but also because they have different neighbors so the normalize term (denominator) will be quite difference.
Then, the meta-path based embedding of node i can be aggregated by the neighbor's projected features with the corresponding coefficients as follows: where z Φ i is the learned embedding of node i for the meta-path Φ. To better understand the aggregating process of node-level, we also give a brief explanation in Figure 3 (a). Every node embedding is aggregated by its neighors. Since the attention weight α Φ i j is generated for single meta-path, it is semantic-specific and able to caputre one kind of semantic information.
Since heterogeneous graph present the property of scale free, the variance of graph data is quite high. To tackle the above challenge, we extend node-level attention to multihead attention so that the training process is more stable. Specifically, we repeat the nodelevel attention for K times and concatenate the learned embeddings as the semantic-specific embedding: Given the meta-path set {Φ 0 , Φ 1 , . . . , Φ P }, after feeding node features into node-level attention, we can obtain P groups of semanticspecific node embeddings, denoted as Z Φ 0 , Z Φ 1 , . . . , Z Φ P .

Semantic-level Attention
Generally, every node in a heterogeneous graph contains multiple types of semantic information and semantic-specific node embedding can only reflect node from one aspect. To learn a more comprehensive node embedding, we need to fuse multiple semantics which can be revealed by meta-paths. To address the challenge of meta-path selection and semantic fusion in a heterogeneous graph, we propose a novel semantic-level attention to automatically learn the importance of different meta-paths and fuse them for the specific task. Taking P groups of semantic-specific node embeddings learned from node-level attention as input, the learned weights of each meta-path (β Φ 0 , β Φ 1 , . . . , β Φ P ) can be shown as follows: Here att sem denotes the deep neural network which performs the semantic-level attention. It shows that the semantic-level attention can capture various types of semantic information behind a heterogeneous graph.
To learn the importance of each meta-path, we first transform semantic-specific embedding through a nonlinear transformation (e.g., one-layer MLP). Then we measure the importance of the semantic-specific embedding as the similarity of transformed embedding with a semantic-level attention vector q. Furthermore, we average the importance of all the semantic-specific node embedding which can be explained as the importance of each meta-path. The importance of each meta-path, denoted as w Φ i , is shown as follows: where W is the weight matrix, b is the bias vector, q is the semanticlevel attention vector. Note that for the meaningful comparation, all above parameters are shared for all meta-paths and semantic-specific embedding. After obtaining the importance of each meta-path, we normalize them via softmax function. The weight of meta-path Φ i , denoted as β Φ i , can be obtained by normalizing the above importance of all meta-paths using softmax function, which can be interpreted as the contribution of the meta-path Φ i for specific task. Obviously, the higher β Φ i , the more important meta-path Φ i is. Note that for different tasks, meta-path Φ i may has different weights. With the learned weights as coefficients, we can fuse these semantic-specific embeddings to obtain the final embedding Z as follows: To better understand the aggregating process of semantic-level, we also give a brief explanation in Figure 3 (b). The final embedding is aggregated by all semantic-specific embedding. Then we can apply the final embedding to specific tasks and design different loss fuction. For semi-supervised node classification, we can minimize the Cross-Entropy over all labeled node between the ground-truth and the prediction: where C is the parameter of the classifier, Y L is the set of node indices that have labels, Y l and Z l are the labels and embeddings of labeled nodes. With the guide of labeled data, we can optimize the proposed model via back propagation and learn the embeddings of nodes. The overall process of HAN in shown in Algorithm 1.

Analysis of the Proposed Model
Here we give the analysis of the proposed HAN as follows: • The proposed model can deal with various types of nodes and relations and fuse rich semantics in a heterogeneous graph. The information can transfer from one kind of nodes to another kind of nodes via diverse relation. Benefitted from such a heterogeneous graph attention network, different types of node embedding can enhance the mutual integration, mutual promotion and mutual upgrade.
• The proposed HAN is highly efficient and can be easily parallelized. The computation of attention can compute individually across all nodes and meta-paths. Given a meta-path Φ, the time complexity of node-level attention is O(V Φ F 1 F 2 K + E Φ F 1 K), where K is the number of attention head, V Φ is the number of nodes, E Φ is the number of meta-path based node pairs, F 1 and F 2 are the numbers The node-level attention weight α , The semantic-level attention weight β .
Calculate the semantic-specific node embedding Concatenate the learned embeddings from all attention Calculate the weight of meta-path β Φ i ; 14 Fuse the semantic-specific embedding Z ← P i=1 β Φ i · Z Φ i ; 15 end 16 Calculate Cross-Entropy L = − l ∈Y L Y l ln(C · Z l ) ; 17 Back propagation and update parameters in HAN; 18 return Z , α, β.
of rows and columns of the transformation matrix, respectively. The overall complexity is linear to the number of nodes and meta-path based node pairs. The proposed model can be easily parallelized, because the node-level and semantic-level attention can be parallelized across node paris and meta-paths, respectively. The overall complexity is linear to the number of nodes and meta-path based node pairs.
• The hierarchical attention is shared for the whole heterogeneous graph which means the number of parameters is not dependent on the scale of a heterogeneous graph and can be used for the inductive problems [14]. Here inductive means the model can generate node embeddings for previous unseen nodes or even unseen graph.
• The proposed model has potentionally good interpretability for the learned node embedding which is a big advantage for heterogeneous graph analysis. With the learned importance of nodes and meta-paths, the proposed model can pay more attention to some meaningful nodes or meta-paths for the specific task and give a more comprensive description of a heterogeneous graph. Based on the attention values, we can check which nodes or meta-paths make the higher (or lower) contributions for our task, which is beneficial to analyze and explain our results.

EXPERIMENTS 5.1 Datasets
The detailed descriptions of the heterogeneous graph used here are shown in Table 2.
• DBLP 2 . We extract a subset of DBLP which contains 14328 papers (P), 4057 authors (A), 20 conferences (C), 8789 terms (T). The authors are divided into four areas: database, data mining, machine learning, information retrieval. Also, we label each author's research area according to the conferences they submitted. Author features are the elements of a bag-of-words represented of keywords. Here we employ the meta-path set {APA, APCPA, APTPA} to perform experiments.
• ACM 3 . We extract papers published in KDD, SIGMOD, SIG-COMM, MobiCOMM, and VLDB and divide the papers into three classes (Database, Wireless Communication, Data Mining). Then we construct a heterogeneous graph that comprises 3025 papers (P), 5835 authors (A) and 56 subjects (S). Paper features correspond to elements of a bag-of-words represented of keywords. We employ the meta-path set {PAP, PSP} to perform experiments. Here we label the papers according to the conference they published.
• IMDB. Here we extract a subset of IMDB which contains 4780 movies (M), 5841 actors (A) and 2269 directors (D). The movies are divided into three classes (Action, Comedy, Drama) according to their genre. Movie features correspond to elements of a bag-ofwords represented of plots. We employ the meta-path set {MAM, MDM} to perform experiments.

Baselines
We compare with some state-of-art baselines, include the (heterogeneous) network embedding methods and graph neural network based methods, to verfify the effectiveness of the proposed HAN. To verify the effectiveness of our node-level attention and semantic-level attention, respectively, we also test two variants of HAN.
• DeepWalk [23]: A random walk based network embedding method which designs for the homogeneous graphs. Here we ignore the heterogeneity of nodes and perform DeepWalk on the whole heterogeneous graph.
• ESim [26]: A heterogeneous graph embedding method which can capture semantic information from multiple meta-paths. Because it is difficult to search the weights of a set of meta-paths, we assign the weights learned from HAN to ESim.
• metapath2vec [7]: A heterogeneous graph embedding method which performs meta-path based random walk and utilizes skip-gram to embed the heterogeneous graphs. Here we test all the meta-paths for metapath2vec and report the best performance.
• HERec [27]: A heterogeneous graph embedding method which designs a type constraint strategy to filter the node sequence and utilizes skip-gram to embed the heterogeneous graphs. Here we test all the meta-paths for HERec and report the best performance.
• GCN [18]: It is a semi-supervised graph convolutional network that designed for the homogeneous graphs. Here we test all the meta-paths for GCN and report the best performance.
• GAT [35]: It is a semi-supervised neural network which considers the attention mechanism on the homogeneous graphs. Here we test all the meta-paths for GAT and report the best performance.
• HAN nd : It is a variant of HAN, which removes node-level attention and assigns the same importance to each neighbor.
• HAN sem : It is a variant of HAN, which removes the semanticlevel attention and assigns the same importance to each meta-path.
• HAN: The proposed semi-supervised graph neural network which employs node-level attention and semantic-level attention simultaneously.

Implementation Details
For the proposed HAN, we randomly initialize parameters and optimize the model with Adam [17]. For the proposed HAN, we set the learning rate to 0.005, the regularization parameter to 0.001, the dimension of the semantic-level attention vector q to 128, the number of attention head K to 8, the dropout of attention to 0.6. And we use early stopping with a patience of 100, i.e. we stop training if the validation loss does not decrease for 100 consecutive epochs. To make our experiments repeatable, we make our dataset and codes publicly available at website 4 .For GCN and GAT, we optimize their parameters using the validation set. For semi-supervised graph neural network, including GCN, GAT and HAN, we split exactly the same training set, validation set and test set to ensure fairness. For random walk based methods include DeepWalk, ESim, metapath2vec, and HERec, we set window size to 5, walk length to 100, walks per node to 40, the number of negative samples to 5. For a fair comparison, we set the embedding dimension to 64 for all the above algorithms.

Classification
Here we employ KNN classifier with k = 5 to perform node classification. Since the variance of graph-structured data can be quite high, we repeat the process for 10 times and report the averaged Macro-F1 and Micro-F1 in Table 3.
Based on Table 3, we can see that HAN achieves the best performance. For traditional heterogeneous graph embedding method, ESim which can leverage multiple meta-paths performs better than metapath2vec. Generally, graph neural network based methods which combine the structure and feature information, e.g., GCN and GAT, usually perform better. To go deep into these methods, compared to simply average over node neighbors, e.g., GCN and HAN nd , GAT and HAN can weigh the information properly and improve the performance of the learned embedding. Compared to GAT, the proposed HAN, which designs for heterogeneous graph, captures the rich semantics successfully and shows its superiority. Also, without nodelevel attention (HAN nd ) or semantic-level attention (HAN sem ), the performance becomes worse than HAN, which indicates the importance of modeling the attention mechanism on both of the nodes and semantics. Note that in ACM and IMDB, HAN improves classification results more significantly than in DBLP. Mainly because APCPA is the much more important than the rest meta-paths. We will explain this phenomenon in Section 5.7 by analyzing the semanticlevel attention.
Through the above analysis, we can find that the proposed HAN achieves the best performance on all datasets. The results demonstrate that it is quite important to capture the importance of nodes and meta-paths in heterogeneous graph analysis.

Clustering
We also conduct the clustering task to evaluate the embeddings learned from the above algorithms. Once the proposed HAN trained, we can get all the node embedding via feed forward. Here we utilize the KMeans to perform node clustering and the number of clusters K is set to the number of classes. We use the same ground-truth as in node classification. And we adopt NMI and ARI to assess the quality of the clustering results. Since the performance of KMeans is affected by initial centroids, we repeat the process for 10 times and report the average results in Table 4.
As can be seen in Table 4, we can find that HAN performs consistently much better than all baselines. Also, graph neural network based algorithms usually achieve better performance. Besides, without distinguishing the importance of nodes or meta-paths, metap-ath2vec and GCN cannot perform well. With the guide of multiple meta-paths, HAN performs significantly better than GCN and GAT. On the other hand, without node-level attention (HAN nd ) or semantic-level attention (HAN sem ), the performance of HAN has shown various degrees of degeneration. It demonstrates that via assigning the different importance to nodes and meta-paths, the proposed HAN can learn a more meaningful node embedding.
Based on the above analysis, we can find that the propsed HAN can give a comprehensive description of heterogeneous graph and achieve a significant improvements.

Analysis of Hierarchical Attention Mechanism
A salient property of HAN is the incorporation of the hierarchical mechanism, which takes the importance of node neighbors and meta-paths into consideration in learning representative embedding. Recall that we have learned the node-level attention weight α Φ i j and the semantic-level attention weight β Φ i . To better understand the importance of the neighbors and meta-paths, we provide a detailed analysis on the hierarchical attention mechanism.
Analysis of node-level attention. As mentioned before, given a specific task, our model can learn the attention values between nodes and its neighbors in a meta-path. Some important neighbors which are useful for the specific task tend to have larger attention values. Here we take the paper P831 5 in ACM dataset as an illustrative example. Given a meta-path Paper-Author-Paper which describes the co-author of different papers, we enumerate the meta-path based neighbors of paper P831 and their attention values are shown in Figure 4. From Figure 4(a), we can see that P831 connects to P699 6 and P133 7 , which all belong to Data Mining; conects to P2384 8 and  Figure 4(b), we can see that paper P831 gets the highest attention value from node-level attention which means the node itself plays the most important role in learning its representation. It is reasonable because all information supported by neighbors are usually viewed as a kind of supplementary information. Beyond itself, P699 and P133 get the second and third largest attention values. This is because P699 and P133 also belong to Data Mining and they can make significant contribution to identify the class of P831. The rest neighbors get minor attention values that because they do not belong to Data Mining and cannot make important contribution to identify the P831's class. Based on the above analysis, we can see that the node-level attention can tell the difference among neighbors and assign higher weights to some meaningful neighbors.
Analysis of semantic-level attention. As mentioned before, the proposed HAN can learn the importance of meta-paths for the specific task. To verify the ability of semantic-level attention, taking DBLP and ACM as examples, we report the clustering results (NMI) of single meta-path and corresponding attention values in Figure 5. Obviously, there is a positive correlation between the performance of a single meta-path and its attention value. For DBLP, HAN gives APCPA the largest weight, which means that HAN considers the APCPA as the most critical meta-path in identifying the author's research area. It makes sense because the author's research area and the conferences they submitted are highly correlated. For example, some natural language processing researchers mainly submit their papers to ACL or EMNLP, whereas some data mining researchers may submit their papers to KDD or WWW. Meanwhile, it is difficult for APA to identify the author's research area well. If we treat these meta-paths equally, e.g., HAN sem , the performance will drop significantly. Based on the attention values of each meta-path, we can find that the meta-path APCPA is much more useful than APA and APTPA. So even the proposed HAN can fuse them, APCPA still plays a leading role in identifying the author's research area while APA and APTPA do not. It also explains why the performance of HAN in DBLP may not be as significant as in ACM and IMDB. We get the similar conclusions on ACM. For ACM, the results show that HAN gives the most considerable weight to PAP. Since the performance of PAP is slightly better than PSP, so HAN sem can achieve good performance by simple average operation. We can see that semantic-level attention can reveal the difference between these meta-paths and weights them adequately. 9 Daniel Barbara, Tomasz Imielinski. Sleepers and Workaholics: Caching Strategies in Mobile Environments, VLDB'95 10 Hector Garcia-Holina, Daniel Barbara. The cost of data replication, SIGCOMM'81

Visualization
For a more intuitively comparation, we conduct the task of visualization, which aims to layout a heterogeneous graph on a low dimensional space. Specifically, we learn the node embedding based on the proposed model and project the learned embedding into a 2-dimensional space. Here we utilize t-SNE [21] to visualize the author embedding in DBLP and coloured the nodes based on their research areas.
From Figure 6, we can find that GCN and GAT which design for the homogeneous graphs do not perform well. The authors belong to different research areas are mixed with each other. Metapath2vec performs much better than the above homogeneous graph neural networks. It demonstrates that the proper meta-path(e.g., APCPA) can make a significant contribution to heterogeneous graph analysis. However, since metapath2vec can only take only one meta-path into consideration, the boundary is still blurry. From Figure 6, we can see that the visualization of HAN peform best. With the guide of multiple meta-paths, the embedding learned by HAN has high intra-class similarity and separates the authors in different research area with distinct boundaries.

Parameters Experiments
In this section, we investigate the sensitivity of parameters and report the results of clustering (NMI) on ACM dataset with various parameters in Figure 7.
• Dimension of the final embedding Z. We first test the effect of the dimension of the final embedding Z. The result is shown in Figure 7(a). We can see that with the growth of the embedding dimension, the performance raises first and then starts to drop slowly. The reason is that HAN needs a suitable dimension to encode the semantics information and larger dimension may introduce additional redundancies.
• Dimension of semantic-level attention vector q. Since the ability of semantic-level attention is affected by the dimension of the semantic-level attention vector q, we explore the experimental results with various dimension. The result is shown in Figure 7(b). We can find that the performance of HAN grows with the dimension of semantic-level attention vector and achieves the best performance when the dimension of q is set to 128. After that, the performance of HAN starts to degenerate which may because of overfitting.
• Number of attention head K. In order to check the impact of multihead attention, we explore the performance of HAN with various number of attention head. The result is shown in Figure  7(c). Note that the multihead attention is removed when the number of attention head is set to 1. Based on the results, we can find   that the more number of attention head will generally improve the performance of HAN. However, with the change of attention head, the performance of HAN improve only slightly. Meanwhile, we also find that multihead attention can make the training process more stable.

CONCLUSION
In this paper, we tackle several fundamental problems in heterogeneous graph analysis and propose a semi-supervised heterogeneous graph neural network based solely on attention mechanism. The proposed HAN can capture complex structures and rich semantics behind heterogeneous graph. The proposed model leverages nodelevel attention and semantic-level attention to learn the importance of nodes and meta-paths, respectively. Meanwhile, the proposed model utilizes the structural information and the feature information in a uniform way. Experimental results include classification and clustering demonstrate the effectiveness of HAN. By analyzing the learned attention weights include both node-level and semantic-level, the proposed HAN has proven its potentially good interpretability.