{'fraction_non_alphanumeric': 0.037650095387723036, 'fraction_numerical': 0.012961508248232521, 'mean_word_length': 4.653449643140365, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
CARL: Content-Aware Representation Learning for Heterogeneous Networks

Heterogeneous networks not only present a challenge of heterogeneity in the types of nodes and relations, but also the attributes and content associated with the nodes. While recent works have looked at representation learning on homogeneous and heterogeneous networks, there is no work that has collectively addressed the following challenges: (a) the heterogeneous structural information of the network consisting of multiple types of nodes and relations; (b) the unstructured semantic content (e.g., text) associated with nodes; and (c) online updates due to incoming new nodes in growing network. We address these challenges by developing a Content-Aware Representation Learning model (CARL). CARL performs joint optimization of heterogeneous SkipGram and deep semantic encoding for capturing both heterogeneous structural closeness and unstructured semantic relations among all nodes, as function of node content, that exist in the network. Furthermore, an additional online update module is proposed for efficiently learning representations of incoming nodes. Extensive experiments demonstrate that CARL outperforms state-of-the-art baselines in various heterogeneous network mining tasks, such as link prediction, document retrieval, node recommendation and relevance search. We also demonstrate the effectiveness of the CARL's online update module through a category visualization study.


INTRODUCTION
Heterogeneous information networks (HetNets) [16,17], e.g., academic networks, encode rich information through multi-typed nodes, relationships, and attributes or content associated with nodes. For example, the academic networks can represent humanhuman relationship (authors), human-object relationship (authorpaper or author-venue or author-organization), and object-object relationship (paper-paper, paper-venue, paper-organization). The nodes in this case (human and object) can carry attributes or semantic content (such as paper abstract). Given the multi-typed nodes, relationships, and content at the nodes, feature engineering Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request   has presented a unique challenge for network mining tasks such as relation mining [15,23], relevance search [7,16], personalized recommendation [9,14,27]. The typical feature engineering activity responds to the requirements of the network mining task for an application, requiring both a domain understanding and large exploratory search space for possible features. Not only this is expensive, but it also may not result in optimal performance. To that end, we ask the question: Can we generalize the feature engineering activity through representation learning on HetNets that addresses the complexity of multi-typed data in HetNets? With the advent of deep learning, significant effort has been devoted to network representation learning in the last few years, starting with a focus on homogeneous networks [5,12,20] and more recently on HetNets [4]. The underlying theme of the models developed in these works is to automate the discovery of useful node latent features that can be further utilized in various network mining problems such as link prediction and node recommendation. However, these methods are limited in truly addressing the challenges of HetNets: • (C1) HetNets include multiple types of nodes and relations. For example, in Figure 1(a), an academic network involves three types of nodes, i.e., author, paper and venue, which are connected by three types of relations, i.e., author-write-paper, paper-citepaper and paper-publish-venue. Most of the previous models (e.g., Deepwalk and node2vec) employ homogeneous language models which make application to HetNets difficult. Thus challenge 1 is: how to extend homogeneous language model to heterogeneous network representation learning for maintaining structural closeness among multiple types of nodes and relations? We build on our prior work metapath2vec [4] for this. • (C2) HetNets include both structural content (e.g., node type and relation connection) and unstructured semantic content (e.g., text). For example, in Figure 1(a), paper in academic network connects to author & venue and contains semantic text. The current models purely depend on structural content yet can not leverage unstructured content to infer semantic relations that are far away in network. To be more specific (as we will show in Section 4.5), given query author "Jure Leskovec", conventional techniques (e.g., node2vec and metapath2vec) tend to return authors who collaborated with "Jure" due to structural relations bridged by paper or return authors who are different from "Jure" in specific research interests due to structural relations bridged by venue. Thus challenge 2 is: how to effectively incorporate unstructured content of nodes into a representation learning framework for capturing both structural closeness and unstructured semantic relations among all nodes? • (C3) HetNets can grow with time. Current models are not able to handle this due to lack of an update strategy and it is impractical to re-run the model for each new node. For example, in Figure  1(b), new author A 5 co-authors with A 1 and A 4 on new paper P 5 after a given time split T. Thus challenge 3 is: how to efficiently learn representations of new nodes in a growing network?
Our proposed method CARL, a content-aware representation learning model for HetNets, addresses these challenges. Specifically, first, we develop a heterogeneous SkipGram model to maintain structural closeness among multiple types of nodes and relations. Next, we design two effective ways based on deep semantic encoding to incorporate unstructured content (i.e., text) of some types of nodes into heterogeneous SkipGram for capturing semantic relations. The negative sampling technique and the walk sampling based strategy are utilized to optimize and train the proposed models. Finally, we develop an online update module to efficiently learn representation of each new node by using its relations with existing nodes and the learned node representations.
To summarize, the main contributions of our work are: • We formalize the problem of content-aware representation learning in HetNets and develop a model, i.e., CARL, to solve the problem. CARL performs joint optimization of heterogeneous SkipGram and deep semantic encoding. • We design the corresponding optimization strategy and training algorithm to effectively learn node representations. The output representations are further utilized in various HetNet mining tasks, such as link prediction, document retrieval, node recommendation and relevance search, which demonstrate the superior performance of CARL over state-of-the-art baselines. • We propose an update module in CARL to handle growing networks and conduct the category visualization study to show the effectiveness of this module.

PROBLEM DEFINITION
We first introduce the concepts of HetNets and random & metapath walks, then formally define the problem of content-aware representation learning in HetNets. For example, in Figure 2(b), the academic network can be seen as a HetNet. The set of node types O V includes author (A), paper (P) and venue (V). The set of link types R E includes author-write-paper, paper-cite-paper and paper-publish-venue. Definition 2.2. (Random Walk) A random walk [5] is defined as a node sequence S v 0 = {v 0 , v 1 , v 2 , ..., v L−1 } wherein the i-th node v i−1 in the walk is randomly selected from the neighbors of its predecessor v i−2 . [4] in Het-Net is defined as a random walk guided by a specific meta-path scheme with the form of P ≡ o 1 and r = r 1 * r 2 · · · * r m−1 represents a compositional relation between relation types r 1 and r m . Each meta-path walk recursively samples a specific P until it meets the given length.  Heterogeneous Networks) Given a HetNet with both structural and unstructured content at each node, the task is to design a model to learn a d-dimensional feature representations θ ∈ R |V |×d (d ≪ |V |), which can encode both structural closeness and unstructured semantic relations. Furthermore, the model is able to efficiently infer representation θ ′ v ′ ∈ R d for each new node v ′ by using the learned representations θ .
For example, in the network of Figure 1, author and venue nodes contain structural content, i.e., node id, node type as well as link relations with others, and paper node contains both structural content and unstructured semantic content, e.g., abstract text. The output θ denotes representations of all existing nodes via the same latent space, which can be further utilized in various HetNet mining tasks. Besides, the learned representation θ ′ v ′ of each new node v ′ can benefit different tasks for v ′ such as category assignment.

CARL FRAMEWORK
We present the framework of content-aware representation learning which will address the three challenges described in Section 1.

Heterogeneous Network Embedding (C1)
Inspired by word2vec [10] for learning distributed representation of words in corpus, Deepwalk [12] and node2vec [5] leverage Skip-Gram and random walks to learn node representations. However, we argued that those techniques focus on homogeneous networks and proposed metapath2vec [4] for HetNets by feeding meta-path walks to SkipGram. Similar to metapath2vec, we formulate the heterogeneous network representation learning as heterogeneous SkipGram (HSG) to address challenge C1. Specifically, given a Het- , the objective is to maximize the likelihood of each type of context node given the input node v: where θ contains the representations of all nodes and N t (v) is the set of t-type context node of v which can be collected in different ways such as one-hop neighbors or surrounding neighbors in random walks. For example, in Figure 2(b), A 3 is structurally close to other authors (e.g., A 1 & A 4 ), papers (e.g., P 2 & P 4 ) and venues (e.g.,

Incorporating Deep Semantic Encoder (C2)
The objective o 1 formulates structural closeness but ignores unstructured semantic relations. To address challenge C2, we design two ways to incorporate unstructured content of some types of nodes into heterogeneous network representation learning.

HSG with Unstructured Semantic Regularization (HSG-SR).
One way is to tightly join HSG with the conditional probability of semantic constraint, leading to unstructured semantic regularization onto objective o 1 . The objective is defined as: where V S is the set of nodes with unstructured semantic content, Y v represents unstructured content of node v and Φ are parameters of a deep semantic encoder that will be described later. The conditional probability p(v c |v; θ ) is defined as the heterogeneous softmax function: . For example, in the network of Figure 2, V S in o 2 is the set of papers and the formulation involves four kinds of representations, i.e., author representations, venue representations, paper representations and paper semantic representations. Notice that, there are two kinds of paper representations and we will use paper semantic representation for evaluation in Section 4.

Unstructured Semantic Enhanced HSG (SE-HSG).
Another way is to concatenate the output of deep semantic encoder with the input of HSG, leading to unstructured semantic enhancement onto objective o 1 . The objective is defined as: where Y is the set of all unstructured semantic content of V S . The conditional probability p(v c |v; θ ; Y ; Φ) is defined as the semantic enhanced heterogeneous softmax function:

Unstructured Semantic Content
Encoder. Both objectives o 2 and o 3 involve deep semantic encoding architecture. To encode unstructured content of some types of nodes into fixed length representations E ∈ R |V S |×d , we introduce gated recurrent units (GRU), a specific type of recurrent neural network, which has been widely adopted for many applications such as machine translation [3]. Figure 2(a) gives an illustrative example of this encoder for papers in the academic network. To be more specific, each paper's abstract is represented as a sequence of words: {w 1 , w 2 , · · · , w t max }, followed by the word embeddings sequence: {x 1 , x 2 , · · · , x t max } trained by word2vec [10], where t max is the maximum length of text. For each step t with the input word embedding x t and previous hidden state vector h t −1 , the current hidden state vector h t is updated by h t = GRU(x t , h t −1 ), where the GRU module is defined as: where σ is the sigmoid function, A and B are parameter matrices of GRU network (i.e., Φ in objectives o 2 and o 3 includes A and B), operator • denotes element-wise multiplication, z t and r t are update gate vector and reset gate vector, respectively. The GRU network encodes word embeddings to deep semantic embeddings h ∈ R t max ×d , which is concatenated with a mean pooling layer to obtain the general semantic representation of paper. All of these steps construct the deep semantic encoder f . We have also explored other encoding architectures such as LSTM, bidirectional GRU and attention-based GRU, and obtain similar results. Thus we choose GRU since it has a concise structure and reduce training time.

Model Optimization and Training
We leverage the negative sampling technique [10] to optimize model and introduce the walk sampling based strategy for model training.
where M is the negative sample size and P t (v c ′ ) is the pre-defined sampling distribution w.r.t. the t-type node. In our case, M makes little impact on the performance of proposed models. Thus we set M = 1 and obtain the cross entropy loss for optimization: where and f is the deep semantic encoder. Therefore we rewrite objective o 2 as: where γ is a trade-off factor and T wal k denotes the set of triplets ⟨v, v c , v c ′ ⟩ collected by walk sampling on HetNet, which will be described later. Besides, T S t r i is the set of nodes with unstructured content in each triplet of T wal k for semantic regularization. That is, max {|T S t r i |} = 3. 3.3.2 Optimization of SE-HSG. Similar to HSG-SR, the loga- As in HSG-SR, T wal k is the set of triplets ⟨v, v c , v c ′ ⟩ collected from walk sequences on HetNet.

Model
Training. Both optimized objectives o 2 and o 3 are accumulated on set T wal k . Similar to Deepwalk, node2vec and metapath2vec, we design a walk sampling strategy to generate T wal k . Specifically, first, we uniformly generate a set of random walks or meta-path walks S in HetNet. Then, for each node v in That is, v's neighbors within distance τ in S i . For example, in Figure  2(c), the context node of A 3 (τ = 2) in the sample walk are V 1 , P 2 , P 4 and V 3 . Finally, for each v c , we sample a negative node v c ′ with the same node type of v c according to Furthermore, we design a minibatch based Adam Optimizer [8] to train the model. Specifically, at each iteration, we sample a mini-batch of triplets in T walk and accumulate the objective according to equation (8) or (10)  until the change between two consecutive iterations is sufficiently small. Figure 2(c) shows an illustration of the framework of HSG-SR and SE-HSG on the academic network. The output representations θ and E can be utilized in various HetNet mining tasks, as we will show in Section 4.

Online Update for New Nodes (C3)
The optimization and training strategies in previous section learn representations of all existing nodes but they cannot be employed in an online situation. Considering the growing property of Het-Nets, we aim to design an online update module to efficiently learn representation of each new node and address challenge C3.

Representation of New Node with Unstructured Semantic
Content. As we introduce deep architecture f to encode semantic representations of nodes with unstructured content, the optimized parameters Φ * can be directly applied to infer representation of such new node without an extra training step. That is, is the text content of v ′ and f * is the learned semantic encoder with Φ * . For example, in the academic network, we can use the learned paper semantic encoder to infer the representation of each new paper.

Representation of New Node without Unstructured Semantic
Content. Inspired by eALS [6] for online recommendation, we make a reasonable assumption that each new node should not change the learned node representations too much from a global perspective. Accordingly, for each incoming node v ′ without unstructured content, we leverage objective o 3 to formulate the following objective: where θ ′ denotes representation of v ′ , Θ * is the learned node representations via SE-HSG, i.e., That is, we take Θ * as constant values in o 4 . In addition, we leverage meta-path walks rooted at v ′ to collect triplet set T v ′ walk for the training process of update module. Specifically, Figure 3 gives an illustrative example of online update for new author in academic network. First, we randomly sample a number of "APVPA" meta-path walks rooted at new author node A 5 . The length of each walk equals the window distance τ (τ = 4) and nodes in the walk are context node of A 5 , e.g., P 5 , V 3 , P 3 and A 2 . For each context node v c , we randomly sample a negative node v c ′ with the same node type as v c . Furthermore, the SGD optimizer is utilized to repeatly update θ ′ for each triplet in T v ′ walk until the change between two consecutive iterations is sufficiently small. The proposed update module is efficient since only simple meta-path walk sampling and limited update steps on new node representation are performed.
In the experimental evaluation, we only consider new author who writes new paper at a venue that exists in network, making the "APVPA" meta-path walk always feasible.

EXPERIMENTS
In this section, we conduct extensive experiments with the aim of answering the following research questions: • (RQ1) How does CARL perform vs. state-of-the-art network representation learning models for different HetNet mining tasks, such as link prediction (RQ1-1), document retrieval (RQ1-2) and node recommendation (RQ1-3)? In addition, how do hyperparameters impact CARL's performance in each task? • (RQ2) What is the performance difference between CARL and baselines in relevance search w.r.t. each task in RQ1? • (RQ3) What is the performance of CARL's online update module on the task for new nodes such as category assignment?
Notice that, although our model can be applied to or modified for different HetNets, we focus on experiments on the academic HetNet due to data availability.

Experimental Setup
4.1.1 Data. We use the AMiner computer science dataset [21], which is publicly available 1 . To avoid noise, we remove the papers published in venues (e.g., workshop) with limited publications and the instances without abstract text. In addition, topic of each area changes over time. For example, according to our analysis of the data, the most popular topics in data mining change from web mining and clustering (1996∼2005) to network mining and learning (2006∼2015). To make a thorough evaluation of CARL and verify its effectiveness for networks in different decades, we independently conduct experiments on two datasets, i.e., AMiner-I (1996∼2005) and AMiner-II (2006∼2015). As a result, AMiner-I contains 160,713 authors, 111,409 papers and 150 venues, AMiner-II contains 571,693 authors, 483,449 papers and 492 venues. The structure of the academic network used in this work is shown in Figure 1.

Comparison Baselines.
We compare CARL with four stateof-the-art models, i.e., Deepwalk [12], LINE [20], node2vec [5] and metapath2vec [4]. Notice that, we use either random walk (rw) or meta-path walk (mw) to collect context node in HSG-SR and SE-HSG, resulting in four variants of CARL: CARL r w H SG−S R , CARL mw H SG−S R , CARL r w S E−H SG and CARL mw S E−H SG . 4.1.3 Reproducibility. For fairness comparison, we use the same representation dimension d = 128 for all models. The window size τ = 7, the number of walks per node N = 10 and the walk length L = 30 are used for Deepwalk, node2vec, metapath2vec and CARL. The same set of parameters is used for CARL's online update module. The size of negative samples M is set to 5 for node2vec, LINE and metapath2vec. In addition, γ = 1.0 for CARL H SG−S R and three meta-path schemes "APA", "APPA" and "APVPA" are jointly used to generate meta-path walks for CARL mw . We employ TensorFlow 1 https://aminer.org/citation to implement all variants of CARL and further conduct them on NVIDIA TITAN X GPU. Code will be available upon publication.

Link Prediction (RQ1-1)
Who will be your academic collaborators? As a response to RQ1-1, we design an experiment to evaluate CARL's performance on the author collaboration link prediction task.

Experimental
Setting. Unlike past work [5] that randomly samples a portion of links for training and uses the remaining for evaluation, we consider a more realistic setting that splits training/test data via a given time stamp T. Specifically, first, the network before T is utilized to learn node representations. Then, the collaboration links before T are used to train a binary logistic classifier. Finally, the collaboration relations after T with equal number of random non-collaboration links are used to evaluate the trained classifier. In addition, only new collaborations among current authors (who appear before T) are considered and duplicated collaborations are removed from evaluation. For example, in Figure 1(b), A 1 and A 4 co-author a new paper P 5 after T. The classifier tends to predict new collaboration between A 1 and A 4 using previous links, e.g., collaboration between A 1 and A 3 . The representation of link is formed by element-wise multiplication between representations of two end nodes. We use Accuracy and F1 score of binary classification as the evaluation metrics. Besides, T is set as 2003/2004 and 2013/2014 for AMiner-I and AMiner-II , respectively.

Results.
The performances of different models are reported in Table 1. According to the table: (a) All variants of CARL perform better than baselines, demonstrating the effectiveness of incorporating unstructured semantic content to learn author representations; (b) CARL mw S E−H SG achieves the best performances in all cases. The average improvements of CARL mw S E−H SG over different baselines range from 10.9% to 41.0% and 6.7% to 30.9% on AMiner-I and AMiner-II , respectively. (c) CARL mw outperforms CARL r w , showing that meta-path walk is better than random walk for collecting context node in CARL. In addition, CARL S E−H SG has better performance than CARL H SG−S R , indicating that concatenating text encoder with heterogeneous SkipGram is more significant than taking text encoding as semantic regularization.

Parameter Sensitivity.
We conduct experiment to analyze the impact of two key parameters, i.e., the window size τ of walk sampling and the representation dimension d. We investigate a specific parameter by changing its value and fixing the others. The prediction results of CARL mw S E−H SG as a function of τ and d on AMiner-II (T = 2013) are shown in Figure 4. We see that: (a) With increasing of τ , accuracy and F1 score increase at first since a larger window means more useful context information. But when τ goes beyond a certain value, performances decrease slowly with τ possibly due to uncorrelated noise. The best τ is around 7; (b) Similar to τ , an appropriate value should be set for d such that the best node representations are learned. The optimal d is around 128.

Document Retrieval (RQ1-2)
Which relevant papers should be retrieved for your query? As a response to RQ1-2, we design an experiment to evaluate CARL's performance on the paper retrieval task.

Experimental
Setting. As in the previous task, the network before T is utilized to learn node representations. The ground truth of relevance is assumed as the co-cited relation between two papers after T. For example, in Figure 1(b), new paper P 5 cites both previous papers P 2 and P 3 . The model tends to retrieve P 2 when querying P 3 or retrieve P 3 when querying P 2 . The relevant score of two papers is defined as the cosine similarity between representations of two papers. We use HitRatio@k as the evaluation metric. Due to large number of candidate papers, we follow the sampling strategy in [26] to reduce evaluation time. Specifically, for each evaluated paper, we randomly generate 100 negative samples for comparison with the true relevant paper. The hit ratio equals 1 if the true relevant paper is ranked in the top-k list of relevant score, otherwise 0. The overall result is the average value of HitRatio@k among all evaluated papers. The duplicated co-cited relations are removed from evaluation and k is set to 10   (b) Representation dimension d Figure 5: Parameter sensitivity in paper retrieval.
that semantic enhanced SkipGram is better than SkipGram with semantic regularization. However, CARL r w has performance close to that of CARL mw , indicating that meta-path walk has little impact for this task. It is reasonable since the paper representations depend on the deep semantic encoder in our model.

Parameter Sensitivity.
Following the same setup in link prediction, we investigate the impact of window size τ and representation dimension d on CARL mw S E−H SG 's performance on AMiner-II (T = 2013), as shown by Figure 5. It can be seen that: (a) The results are little sensitive to τ when τ ≥ 7. As we noted above, paper representations depend on the deep semantic encoder; (b) The dimension d plays significant role on generating paper representations. The best representations are learned when d is around 128 for the paper retrieval task.

Node Recommendation (RQ1-3)
Which venues should be recommended to you? As a response to RQ1-3, we design an experiment to evaluate CARL's performance on the venue recommendation task.

Experimental Setting.
As in the previous two tasks, the network before T is utilized to learn node representations. The ground truth of recommendation is based on author's appearance in venue after T. For example, in Figure 1(b), author A 1 writes  new paper P 5 on V 3 after T, indicating A 1 is likely to accept V 3 recommendation before T. The preference score is defined as the cosine similarity between representations of author and venue. We use Recall@k as the evaluation metric and k is set to 5 or 10. In addition, the duplicated author-venue pairs are removed from evaluation. The reported score is the average value over all evaluated authors.

Results.
The results are reported in Table 3. From the table: (a) All variants of CARL achieve better performance than baselines, showing the benefit of incorporating unstructured semantic content for learning author and venue representations; (b) The average improvements of CARL mw S E−H SG over different baselines are significant, and range from 18.9% to 178.7% and 11.7% to 275.6% on AMiner-I and AMiner-II ; (c) The results of different variants of CARL are close due to relative small recall values. However, we find that CARL r w H SG−S R is the worst among four, indicating both meta-path walk and semantic enhanced SkipGram help improve the performance of CARL in the venue recommendation task. Figure 6 shows the impact of window size τ and feature dimension d on the performance of CARL mw S E−H SG on AMiner-II (T = 2013). Accordingly, CARL mw S E−H SG achieves the best results when τ is around 7 and d is around 128 for the venue recommendation task.

Relevance Search: Case Study (RQ2)
To answer RQ2, we present three case studies of relevance search on AMiner-II (T = 2013) to show the performance differences between CARL mw S E−H SG and baselines. The ranking of each search result is based on the cosine similarity of representations. Table 4 lists the top-5 returned authors for query author "Jure Leskovec" of CARL mw S E−H SG and two baselines, i.e., node2vec and metapath2vec, which achieves relatively better performances in the collaboration prediction task. According to this table: (a) most of returned authors of node2vec have collaboration relations with "Jure" before T, indicating that node2vec highly depends on structural closeness and cannot find relevant authors who are far away from "Jure" in the network; (b) metapath2vec returns some authors (e.g., P. Nguyen) who are different from "Jure" in their specific research interests, illustrating that "APVPA" meta-path walks (used by metapath2vec) may collect context node that are different from target node since it is common that authors bridged by the same venue study different research topics; (c) CARL mw S E−H SG not only returns structurally close authors who have collaboration relations with "Jure" before T but also finds farther authors (e.g., D. Romero) who share similar research interest with "Jure", demonstrating CARL mw S E−H SG captures both structural closeness and unstructured semantic relations for learning author representations. Table 5 lists the top-5 returned papers for query paper "When will it happen?" of CARL mw S E−H SG and the best baseline node2vec in paper retrieval task. From this table: (a) all returned papers of node2vec are written by at least one author in query paper, showing that node2vec only returns structurally close papers but has difficulty finding farther semantically related papers; (b) CARL mw S E−H SG not only returns structurally close papers which have common authors with query paper but also finds semantically related papers without authorship overlapping, showing that CARL mw S E−H SG utilizes both structural content and unstructured semantic content for learning paper representations. Table 6 lists the top-10 returned venues for query author "Chi Wang" of CARL mw S E−H SG and two baselines, i.e., Deepwalk and node2vec, which have relatively better performances on the venue recommendation task. We find that: (a) Deepwalk and node2vec recommend both data mining (e.g., KDD & ICDM) and database (e.g., SIGMOD & PVLDB ) venues to "Chi" since some of his works cite database papers and some of his co-authors focus on database research, which illustrates that both Deepwalk and node2vec return structurally close venues, some of which are not the most suitable ones; (b) most of the venues in CARL mw S E−H SG 's recommendation list belong to data mining related areas, demonstrating that incorporating semantic content helps learn better representations of author and venue.

Category Visualization of New Nodes (RQ3)
The previous experiments and case studies demonstrate the effectiveness of CARL in learning representations of current nodes, i.e., nodes that exist in HetNet before T. As described in Section 3.4, the learned semantic encoder of CARL can be directly applied to infer representation of each incoming node with semantic content.     1  KDD  KDD  KDD  2  CIKM  WSDM  CIKM  3  DASFAA  SIGMOD  ICDM  4  SIGMOD  PVLDB  PAKDD  5  ICDM  WWW  WWW  6  WSDM  ICDM  WWWC  7  ICDE  EDBT  TKDE  8  PVLDB  PAKDD  KAIS  9  WWW  GRC  DASFAA  10 EDBT FPGA WSDM Besides, CARL's online update module can efficiently learn representation of each new node without semantic content. To answer RQ3 and show the effectiveness of learned new node representations, we employ the Tensorflow embedding projector to visualize the new paper representations inferred by paper semantic encoder and new author representations learned by online update module in sequence. Figure 7 shows  is assigned to the area with the majority of his/her publications. We randomly sample 100 new papers/authors of each area. According to Figure 7: (a) The representations of new papers in the same category cluster closely and can be well discriminated from others for both 2D and 3D visualizations, indicating that the semantic encoder achieves satisfactory performance in inferring semantic representations of new papers. Notice that, papers belong to DM have few intersections with the other three since semantic content of few DM papers are quite similar to CV, HCI and Theory papers w.r.t. model, application and theoretical basis, respectively. (b) The representations of new authors in the same category are clearly discriminated from others without intersection for both 2D and 3D visualizations, which demonstrates the effectiveness of online update module in learning new author representations.

RELATED WORK
In the past decade, many works have been devoted to mining Het-Nets for different applications, such as relevance search [2,7,16,28], node clustering [17,18], personalized recommendation [14,26,27]. The network representation learning has gained a lot of attention in the last few years. Some walk sampling based models [4,5,12] have been proposed to learn vectorized node representations that can be further utilized in various tasks in network. Specifically, inspired by word2vec [10] for learning distributed representations of words in text corpus, Perozzi et al. developed the innovative Deepwalk [12] which introduces node-context concept in network (analogy to word-context) and feeds a set of random walks over network (analogy to "sentences") to SkipGram for learning node representations. In order to deal with neighborhood diversity, Grover & Leskovec suggested taking biased random walks (a mixture of BFS and DFS) as the input of SkipGram. More recently, we argued that those models are not able to truly tackle network heterogeneity and proposed metapath2vec [4] for heterogeneous network representation learning by feeding meta-path walks to SkipGram. In addition, many other models have been proposed [1,11,13,19,20,22,24,25], such as PTE [19] for text data embedding, HNE [1] for image-text data embedding and NetMF [13] for unifying network representation models as matrix factorization.
Our work furthers the investigation of network representation learning by developing a content-aware representation learning model CARL for HetNets. Unlike previous models, CARL leverages both structural closeness and unstructured semantic relations to learn node representations, and contains an online update module for learning representations of new nodes.

CONCLUSION
In this paper, we formalize the problem of content-aware representation learning in HetNets and propose a novel model CARL to solve the problem. CARL performs joint optimization of heterogeneous SkipGram and deep semantic encoding for capturing both structural closeness and unstructured semantic relations in a HetNet. Furthermore, an online update module is designed to efficiently learn representations of new nodes. Extensive experiments demonstrate that CARL outperforms state-of-the-art baselines in various HetNet mining tasks, such as link prediction, document retrieval, node recommendation and relevance search. Besides, the online update module achieves satisfactory performance, as reflected by category visualization of new nodes. In the future, we plan to design the dynamic heterogeneous network representation learning models by using time series information of nodes and links.