{'fraction_non_alphanumeric': 0.02954492686324588, 'fraction_numerical': 0.008503598792663106, 'mean_word_length': 4.958326128307107, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
LSCALE: Latent Space Clustering-Based Active Learning for Node Classification

Node classification on graphs is an important task in many practical domains. It usually requires labels for training, which can be difficult or expensive to obtain in practice. Given a budget for labelling, active learning aims to improve performance by carefully choosing which nodes to label. Previous graph active learning methods learn representations using labelled nodes and select some unlabelled nodes for label acquisition. However, they do not fully utilize the representation power present in unlabelled nodes. We argue that the representation power in unlabelled nodes can be useful for active learning and for further improving performance of active learning for node classification. In this paper, we propose a latent space clustering-based active learning framework for node classification (LSCALE), where we fully utilize the representation power in both labelled and unlabelled nodes. Specifically, to select nodes for labelling, our framework uses the K-Medoids clustering algorithm on a latent space based on a dynamic combination of both unsupervised features and supervised features. In addition, we design an incremental clustering module to avoid redundancy between nodes selected at different steps. Extensive experiments on five datasets show that our proposed framework LSCALE consistently and significantly outperforms the stateof-the-art approaches by a large margin.


Introduction
Node classification on graphs has attracted much attention in the graph representation learning area. Numerous graph learning methods [15,11,27,6] have been proposed for node classification with impressive performance, especially on the semi-supervised setting, where labels are required for the classification task.
In reality, labels are often difficult and expensive to collect. To mitigate this issue, active learning aims to select the most informative data points which can lead to better classification performance using the same amount of labelled data. Graph neural networks (GNNs) have been used for some applications such as disease prediction and drug discovery [21,10], in which labels often have to be obtained through costly means such as chemical assays. Thus, these applications motivate research into active learning with GNNs.
In this work, we focus on active learning for node classification on attributed graphs. Recently, a few GNN-based active learning methods [5,9,28,22,13] have been proposed for attributed graphs. However, their performance is still less than satisfactory in terms of node classification. These approaches do not fully utilize the useful representation power in unlabelled nodes and only use unlabelled nodes for label acquisition. For example, AGE [5] and ANRMAB [9] select corresponding informative nodes to label based on the hidden representations of graph convolutional networks (GCNs) and graph structures. These hidden representations can be updated only based on the labelled data. On the other hand, FeatProp [28] is a clustering-based algorithm which uses propagated node attributes to select nodes to label. However, these propagated node attributes are generated in a fixed manner based on the graph structure and node attributes, and are not learnable. In summary, existing approaches do not fully utilize the information present in unlabelled nodes. To utilize the information in unlabelled nodes, a straightforward method is to use features extracted from a trained unsupervised model for choosing which nodes to select. For example, FeatProp can conduct clustering based on unsupervised features for selecting nodes. However, as shown in our experimental results, it still cannot effectively utilize the information in unlabelled nodes for active learning.
Motivated by the limitations above, we propose an effective Latent Space Clustering-based Active LEarning framework (hereafter LSCALE). In this framework, we conduct clustering-based active learning on a designed latent space for node classification. Our desired active learning latent space should have two key properties: 1) low label requirements: it should utilize the representation power from all nodes, not just labelled nodes, thereby obtaining accurate representations even when very few labelled nodes are available; 2) informative distances: in this latent space, intra-class nodes should be closer together, while inter-class nodes should be further apart. This can facilitate clustering-based active selection approaches, which rely on these distances to output a diverse set of query points.
To achieve these, our approach incorporates an unsupervised model (e.g., DGI [26]) on all nodes to generate unsupervised features, which utilizes the information in unlabelled nodes, satisfying the first desired property. In addition, we design a distance-based classifier to classify nodes using the representations from our latent space. This ensures that distances in our latent space are informative for active learning selection, satisfying our second desired property. To select nodes for querying labels, we leverage the K-Medoids clustering algorithm in our latent space to obtain cluster centers, which are the queried nodes. As more labelled data are received, the distances between different nodes in the latent space change based on a dynamic combination of unsupervised learning features and learnable supervised representations. Furthermore, we propose an effective incremental clustering strategy for clustering-based active learning to prevent redundancy during node selection. Existing clustering-based active learning methods like [24,28] only select nodes in multiple rounds with a myopic approach. More specifically, in each round, they apply clustering over all unlabelled nodes and select center nodes for labelling. However, the cluster centers tend to be near the ones obtained in the previous rounds. Therefore, the clustering can select redundant nodes and does not provide much new information in the later rounds. In contrast, our incremental clustering is designed to be aware of the selected nodes in the previous rounds and ensure newly selected nodes are more informative. Our contributions are summarized as follows: -We propose a latent space clustering-based active learning framework (LSCALE) for node classification on attributed graphs. LSCALE contains a latent space with two key properties designed for active learning purposes: 1) low label requirements, 2) informative distances. -We design an incremental clustering strategy to ensure that newly selected nodes are not redundant with previous nodes, which further improves the performance. -We conduct comprehensive experiments on three public citation datasets and two co-authorship datasets. The results show that our method provides a consistent and significant performance improvement compared to the state-of-the-art active learning methods for node classification on attributed graphs.

Problem Definition
In this section, we present the formal problem definition of active learning for node classification. Let G = (V, E) be a graph with node set V and edge set E, where |V | = n and |E| = m. X ∈ R n×d and Y represent the input node attribute matrix and label matrix of graph G, respectively. In particular, each node v ∈ V is associated with a length-d attribute vector x v and a one-hot label vector y v . Given a graph G and its associated attribute matrix X, node classification aims to find a model M which predicts the labels for each node in G such that the loss function L(M|G, X, Y) over the inputs (G, X, Y) is minimized. Furthermore, the problem of active learning for node classification is formally defined as follows. In each step t, given the graph G and the attribute matrix X, an active learning strategy A selects a node subset S t ⊆ U t−1 for querying the labels y i for each node i ∈ S t . After getting the new set of labelled nodes S t , we obtain a set of all labelled nodes L t = S t L t−1 and a set of unlabelled nodes U t = U t−1 \ S t prepared for the next iteration. Then G and X with labels y i of i ∈ L t are used as training data to train a model M at the end of each step t. We define the labelling budget b as the total maximum number of nodes which are allowed to be labelled. The eventual goal is to maximize the performance of the node classification task under the budget b. To achieve this, active learning needs to carefully select nodes for labelling (i.e., choose S t at each step t). The objective is to minimize the loss using all labelled nodes L t at each step t:

Methodology
In this section, we introduce our active learning framework LSCALE for node classification in a top-down fashion. First, we describe the overview and the key idea of LSCALE. Then we provide the details of each module used in LSCALE.
The overview of our latent space clustering-based active learning framework is shown in Figure 1. The most important aspect of our framework is to design a suitable active learning latent space, specifically designed for clustering-based active learning. Motivated by the limitations of previous methods, we design a latent space with two important properties: 1) low label requirements: the latent space representations can be learned effectively even with very few labels, by utilizing the representation power from all nodes (including unlabelled nodes) rather than only labelled nodes; 2) informative distances: in the latent space, distances between intra-class nodes should be smaller than distances between inter-class nodes. With the first property, LSCALE can learn effective node representations throughout the active learning process, even when very few labelled nodes have been acquired. The second property makes distances in our latent space informative with respect to active learning selection, ensuring that clustering-based active selection processes choose a diverse set of query points.
To satisfy the first property, we use an unsupervised learning method to learn unsupervised node representations H ∈ R n×d based on graphs and node attributes, where d is the dimension of representations. After obtaining H, we design a linear distance-based classifier to generate output predictions. In the classifier, we apply a learnable linear transformation on H to obtain hidden representations Z. The distances between nodes are calculated by a dynamic combination of both Z and H, which satisfies the second desired property. Clustering is performed on the latent space using the distances to select informative nodes. We additionally propose an incremental clustering method to ensure that the newly selected nodes are not redundant with the previously selected nodes. In summary, the framework contains a few main components: an unsupervised graph learning method to generate unsupervised node representations.
an active learning latent space with two aforementioned properties: 1) low label requirements, 2) informative distances. an incremental clustering method to select data points as centroids, which we use as the nodes to be labelled, and prevent redundancy during node selection.

Active Learning Latent Space
To facilitate clustering-based active learning, we need a latent space with two desired properties. Therefore, we propose a distance-based classifier for generating representations from supervised signals and a distance function to dynamically consider supervised and unsupervised representations simultaneously.
Distance-based classifier. In our framework, we design a novel distance-based classifier to ensure that distances between nodes in our latent space can facilitate active learning further. Intuitively, a desired property of the latent space is that nodes from different classes should be more separated and nodes with the same class should be more concentrated in the latent space. Thus, it can help clustering-based active learning methods select representative nodes from different classes. To achieve this, we first map unsupervised features H to another set of features Z by a linear transformation: where W c ∈ R d ×l is the trainable linear transformation matrix. l is the dimension of latent representations. Then we define a set of learnable class representations c 1 , c 2 , ..., c K , where K is the number of classes. The distance vector of node i is defined as: where ⊕ is the concatenation operation and || · || 2 is the L 2 norm. The j-th elementŷ ij in the output predictionŷ i of node i is obtained by the softmax function:ŷ For training the classifier, suppose the labelled node set at step t is L t . The cross-entropy loss function for node classification over the labeled node set is defined as: where y ic denotes the c-th element in the label vector y i .
With the guidance of labelled nodes and their labels via backpropagation, we can update the transformation matrix W c and new features Z can capture the supervised information from labelled data. In addition, new features Z allow intra-class nodes more close and inter-class nodes more separate in the feature space. Through this distance-based classifier, the generated feature space allows the clustering-based active selection effectively select a diverse set of query nodes. Distance function. In LSCALE, the distance function determines the distances between nodes in the latent space for further clustering. We define our distance function as: where g(X) is a mapping from node attributes X to new distance features. As previous graph active learning methods do not effectively utilize the unlabelled nodes, we aim to take advantage of unsupervised learning features and supervised information from labelled data. To this end, we combine unsupervised learning features H and supervised hidden representations Z in the distance function.
A straightforward way to combine them is using concatenation of H and Z: Noted that H and Z are in different spaces and may have different magnitudes of row vectors. We define the distance features as follows: where H and Z are l 2-normalized H and Z respectively to make sure they have same Euclidean norms of rows. α can be treated as a parameter for controlling the dynamic combination of unsupervised features and supervised features. Intuitively, Z can be unstable in the early stages as there are relatively few labelled nodes in the training set. So, in the early stages, we would like to focus more on unsupervised features H, which are much more stable than Z. As the number of labelled nodes increases, the focus should be shifted to hidden representations Z in order to emphasize supervised information. Inspired by curriculum learning approaches [2], we set an exponentially decaying weight as follows: where |L t | is the number of labelled nodes at step t. λ can be set as a number close to 1.0, e.g., 0.99. By using this dynamic combination of unsupervised learning features H and supervised hidden representations Z, we eventually construct the latent space g(X) which has the two important properties: 1) low label requirements: it utilizes the representation power from all nodes including unlabelled nodes; 2) informative distances: distances between nodes are informative for node selection. Thus, the latent space can facilitate selecting diverse and representative nodes in the clustering module. Note that FeatProp [28] uses propagated node attributes as representations for calculating distances. The propagated node attributes are fixed and not learnable throughout the whole active learning process, which makes the node selection less effective. In contrast, our latent space is learned based on signals from both labelled and unlabelled data. In addition, it gradually shifts its focus to emphasize supervised signals as we acquire more labelled data.

Clustering Module
At each step, we use the K-Medoids clustering on our latent space to obtain cluster representatives. In K-Medoids, medoids are chosen from among the data points themselves, ensuring that they are valid points to select during active learning. So, after clustering, we directly select these medoids for labelling. This ensures that the chosen centers are well spread out and provide good coverage of the remaining data, which matches the intuition of active learning, since we want the chosen centers to help us classify as much as possible of the rest of the data. At each step t, the objective of K-Medoids is: Besides K-Medoids, common clustering methods used in the previous work are K-Means [5,9] and K-Centers [24]. K-Means cannot be directly used for selecting nodes in active learning as it does not return real sample nodes as cluster representatives. Incremental clustering. Despite these advantages of K-Medoids for active learning on graphs, a crucial drawback is that it is possible to select similar nodes for querying during multiple iterations. That is, newly selected nodes may be close to previously selected ones, making them redundant and hence worsening the performance of active learning. The reason is that the clustering algorithm only generates the representative nodes in the whole representation space without the awareness of previously selected nodes. To overcome this problem, we design an effective incremental clustering algorithm for K-Medoids to avoid selecting redundant nodes.
In our incremental clustering method, the key idea is that fixing previous selected nodes as some medoids can force the K-Medoids algorithm to select additional medoids that are dissimilar with the previous ones. We illustrate our incremental clustering method in Algorithm 1.
After calculating the distances for every node pair (Line 4), incremental K-Medoids is conducted (Line 5 to Line 15). Compared to the original K-Medoids, the most important modification is that only clusters with a medoid, which is not in the previous labelled nodes set (i.e., m ∈ L t−1 ), can update the medoid (Line 10-13). When all the medoids are the same as those in the previous iteration, the K-Medoids algorithm stops and keeps the medoids. For the medoids which are not the previous selected nodes, we put them in selected node set S t , meanwhile we set labelled node set L t and unlabelled node set U t using S t accordingly.

Experiments
The main goal of our experiments is to verify the effectiveness of our proposed framework LSCALE 1 . We design experiments to answer the following research questions:

Algorithm 1: Incremental K-Medoids clustering
Input: the set of previous labelled nodes L t−1 , the set of unlabelled nodes U t−1 as the pool, the budget b t of the current step.  16 Construct selected node set S t using the medoids m ∈ L t−1 ; . Overall performance and effectiveness of unsupervised features: How does LSCALE perform as compared with state-of-the-art graph active learning methods? Is utilizing unsupervised features also helpful for other clustering-based graph active learning methods? -RQ2. Efficiency: How efficient is LSCALE as compared with other methods? -RQ3. Ablation study: Are the designed dynamic feature combination and incremental clustering useful to improve the performance? How does our distance-based classifier affect the performance?
Datasets. To evaluate the effectiveness of LSCALE, we conduct the experiments on Cora, Citeseer [23], Pubmed [20], Coauthor-CS (short as Co-CS) and Coauthor-Physics (short as Co-Phy) [25]. The first three are citation networks while Co-CS and Co-Phy are two co-authorship networks. We describe the datasets in detail and summarize the dataset statistics in Supplement B.1.
Baselines. In the experiments, to show the compatibility with different unsupervised learning methods, we use two variants LSCALE-DGI and LSCALE-MVGRL, which use DGI [26] and MVGRL [12] as the unsupervised learning method, respectively. To demonstrate the effectiveness of LSCALE, we compare two variants with the following representative active learning methods on graphs. Random: select the nodes uniformly from the unlabelled node pool; Uncertainty: select the nodes with the max information entropy according to the cur-  rent model. AGE [5] constructs three different criteria based on graph neural networks to choose a query node. Combining these different criteria with timesensitive variables to decide which nodes to selected for labelling. ANRMAB [9] proposes a multi-arm-bandit mechanism to assign different weights to the different criteria when constructing the score to select a query node. FeatProp [28] performs the K-Medoids clustering on the propagated features obtained by simplified GCN [27] and selects the medoids to query their labels. GEEM [22]: inspired by error reduction, it uses simplified GCN [27] to select the nodes by minimizing the expected error.
As suggested in [5,9,28], AGE, ANRMAB, FeatProp, Random, and Uncertainty use GCNs as the prediction model, which is trained after receiving labelled nodes at each step. GEEM uses the simplified graph convolution (SGC) [27] as the prediction model as mentioned in [22].

Experimental Setting
We evaluate LSCALE-DGI, LSCALE-MVGRL, and other baselines on node classification task with a transductive learning setup, following the experimental setup as in [9,28] for a fair comparison. Dataset splits. For each citation dataset, we use the same testing set as in [15], which contains 1000 nodes. For coauthor datasets, we randomly sample 20% nodes as the testing sets. From the non-testing set in each dataset, we randomly sample 500 nodes as a validation set and fix it for all the methods to ensure a fair comparison. Experiment procedure. In the experiments, we set the budget sizes differently for different datasets and we focus on the "batched" multi-step setting as in [24,28]. Each active learning method is provided a small set of labelled nodes as an initial pool. As in [28], we randomly select 5 nodes regardless of the class as an initial pool. The whole active learning process is as follows: (1) we first train the prediction model with initial labelled nodes. (2) we use the active learning strategy to select new nodes for labelling and add them to the labelled node pool; (3) we train the model based on the labelled nodes again. We repeat Step (2) and Step (3) until the budget is reached and train the model based on the final labelled node pool. For clustering-based methods (i.e., FeatProp and LSCALE), 10 nodes are selected for labelling in each iteration as these methods depend on selecting medoids to label. Hyperparameter settings. For hyperparameters of other baselines, we set them as suggested in their papers. We specify hyperparameters of our methods in Supplement B.2.

Performance Comparison (RQ1)
Overall comparison. We evaluate the performance by using the averaged classification accuracy. We report the results over 20 runs with 10 different random data splits. In Table 1 and 2, we show accuracy scores of different methods when the number of labelled nodes is less than 60. Analysing Table 1

Efficiency Comparison (RQ2)
We empirically compare the efficiency of LSCALE-D with that of four state-ofthe-art methods (i.e., AGE, ANRMAB, FeatProp, and GEEM). Table 3 shows the total running time of these models on different datasets. From Table 3, GEEM has worst efficiency as it trains the simplied GCN model n × K times (K is the number of classes) for selecting a single node. FeatProp and LSCALE-D are much faster than the other methods. The reason is that FeatProp and LSCALE-D both select several nodes in a step and train the classifier once for this step, whereas AGE, ANRMAB and GEEM all select a single node once in a step. Comparing LSCALE-D and FeatProp, LSCALE-D requires less time as the clustering in LSCALE-D is performed in the latent space where the dimension is less than that in the original attribute space used in FeatProp.

Ablation Study (RQ3)
Effectiveness of dynamic feature combination and incremental clustering. We conduct an ablation study to evaluate the contributions of two different components in our framework: dynamic feature combination and incremental clustering. The results are shown in Figure 3. DGI features is the variant without either dynamic combination or incremental clustering, and it only uses features obtained by DGI as distance features for the K-Medoids clustering algorithm. Dynamic Comb uses the dynamic combination to obtain distance features for clustering. LSCALE is the full version of our variant with dynamic feature combination and incremental clustering. It is worth noting that DGI features can be considered as a simple method utilizing unsupervised features. Analysing Figure  3, we have the following observations: -Dynamic Comb generally provides better performance than DGI features, which shows the effectiveness of our dynamic feature combination for distance features. -LSCALE and Dynamic Comb provide no much different performance when the number of labelled nodes is relatively low. However, LSCALE gradually outperforms Dynamic Comb as the number of labelled nodes increases. This confirms that incremental clustering can select more informative nodes by avoiding redundancy between nodes selected at different steps.
In summary, the results verify the effectiveness and necessity of dynamic combination and incremental clustering.
Effectiveness of our distance features g(X). Furthermore, we also qualitatively demonstrate the effectiveness of our distance feature g(X). labelled nodes. Recall that FeatProp uses propagated node attributes as distance features and DGI features are learned using an unsupervised method with unlabelled data. Compared with others, the distance features used in LSCALE-DGI have clearer boundaries between different classes, which satisfies our second desired property (informative distances) and further facilitates selecting informative nodes in the clustering algorithm.
Effectiveness of distance-based classifier. We design a distance-based classifier in LSCALE to ensure that distances are informative in the active learning latent space. To demonstrate the effectiveness of the distance-based classifier, we replace it with a GCN classifier and show the comparison in Table 4. With the distance-based classifier, LSCALE can achieve better performance than that with a GCN classifier. This comparison shows the effectiveness of the designed distance-based classifier.
To further investigate whether the distance-based classifier is also effective for other active learning methods, we change the GCN classifier to our proposed distance-based classifier for FeatProp and present the comparison in Table 5. From Table 5, we note that FeatProp with our distance-based classifier has slightly better performance compared with FeatProp with GCN classifier on all  the datasets. This observation indicates that our distance-based classifier is also effective for other clustering-based active learning methods.

Related Work
Active Learning on Graphs. For active learning on graphs, early works without using graph representations are proposed in [4,19,3], where the graph structure is used to train the classifier and calculate the query scores for selecting nodes. More recent works [8,1] study non-parametric classification models with graph regularization for active learning with graph data. Recent works [5,9,28,22] utilize graph convolutional neural networks (GCNs) [15], which consider the graph structure and the learned embeddings simultaneously. AGE [5] design an active selecting strategy based on a weighted sum of three metrics considering the uncertainty, the the graph centrality and the information density. Improving upon the weight assignment mechanism, ANRMAB [9] designs a multi-armed bandit method with a reward scheme to adaptively assign weights for the different metrics.
Besides the metric-based active selection on graphs, FeatProp [28] uses a clustering-based active learning method, which calculates the distances between nodes based on representations of a simplified GCN model [27] and conducts a clustering algorithm (i.e., K-Medoids) for selecting representative nodes. A recent method GEEM [22] uses a simplified GCN [27] for prediction and maximizes the expected error reduction to select informative nodes to label. Rather than actively selecting nodes and training/testing on a single graph, [13] learns a selection policy on several labelled graphs via reinforcement learning and actively selects nodes using that policy on unlabelled graphs. [16,18] use adversarial learning and meta learning approaches for active learning on graphs. However, even with relatively complicated learning methods, their performance are similar with AGE [5] and ANRMAB [9]. [7] investigates active learning on heterogeneous graphs. [29] considers noisy oracle setting where labels obtained by an oracle can be incorrect. In this work, we focus on the homogeneous single-graph setting like in [5,9,28,22]. To tackle limitations of previous work on this setting, we have presented an effective and efficient framework that can utilize the representation power in unlabelled nodes and achieve better performance under the same labelling budget.

Conclusion
In this paper, we focus on active learning for node classification on graphs and argue that existing methods are still less than satisfactory as they do not fully utilize the information in unlabelled nodes. Motivated by this, we propose LSCALE, a latent space clustering-based active learning framework, which uses a latent space with two desired properties for clustering-based active selection. We also design an incremental clustering module to minimize redundancy between nodes selected at different steps. Extensive experiments demonstrate that our method provides superior performance over the state-of-the-art models. Our work points out a new possibility for active learning on graphs, which is to better utilize the information in unlabelled nodes by designing a feature space more suitable for active learning. Future work could propose new unsupervised methods which are more integrated with active learning process and enhance our framework further.

Supplementary Materials A Symbols
The number of nodes and the number of attributes in G A, D The adjacency matrix and the degree matrix of graph G S t The set of selected nodes at step t U t The set of unlabelled nodes at step t L t The set of labelled nodes at step t X, Y Node attributes and labelŝ Y The model predictions d (vi, vj) The distance between node vi and vj for clustering B More on Experiments

B.1 Datasets
we conduct the experiments on Cora, Citeseer [23], Pubmed [20], Coauthor-CS (short as Co-CS) and Coauthor-Physics (short as Co-Phy) [25]. The first three are citation networks, which are undirected networks and contain unweighted edges among different publications. Node attributes in Cora and Citeseer are provided as bag-of-words features, while node attributes of Pubmed are TF/IDF weighted word vectors. Co-CS and Co-Phy are two co-authorship networks where nodes are authors, which are connected by an edge if they coauthor a paper. Node attributes in these datasets are bag-of-words encoded paper keywords. We summarize the dataset statistics for different datasets in Table 7.

B.2 Hyperparameter settings
For hyperparameters of other baselines, e.g., the number of layers of GCNs, the number of hidden units of GCNs, the optimizer, the learning rate, we set them as suggested in their papers. For the embedding dimensionality d and other hyperparameters of corresponding unsupervised methods in LSCALE-DGI and LSCALE-MVGRL, we set them as suggested in the corresponding paper [26,12]. For the dimensionality l of hidden representation Z, we set as 100 for all datasets. By default, We set λ = 0.99 to determine α at different steps. To train the classifier in LSCALE, we use the Adam [14] optimizer with the learning rate 0.2 and the weight decay 5 × 10 −6 for maximum 300 epochs and early stopping with a window size of 10.

B.3 More Experimental Results
In Figure 4, we focus on how accuracy scores of different methods change as the number of labelled nodes increases on Cora, Citeseer, Co-Phy, and Co-CS. We omit Pubmed in Figure 4 because of space limitations and also the similar performance as other datasets. Comparing AGE and FeatProp on Cora, Co-Phy, and Co-CS, FeatProp initially outperforms AGE. However, AGE outperforms FeatProp as more labelled node are obtained, showing that supervised information is indeed helpful for effectively selecting informative nodes. Recall that FeatProp only uses propagated node attributes without considering supervised information during the node selection process, which makes it less effective as the number of labelled nodes increases on some datasets. GEEM consistently outperforms other state-of-the-art methods on Citeseer and Co-CS. Our two variants LSCALE-DGI and LSCALE-MVGRL generally outperform other methods under different budgets on all datasets, thanks to the proposed dynamic combination of unsupervised learning features and supervised hidden representations.

B.4 Parameter Study
We study the effect of varying the parameters in our method, including the number of hidden units l in our classifier and λ for controlling the dynamic feature combination (Equation 8). The results are demonstrated in Figure 5 and 6. We alter the number of hidden units l among {25, 50, 100, 200, 400} and λ among {0.7, 0.8, 0.9, 0.95, 0.99}. As we can see in Figure 5, changing l has no much difference in terms of performance on Co-CS, while l = 25 performs best on Cora when the number of labelled nodes is less than 60. However, when it is larger than 60, different l generate similar performances on Cora. These results show that our method is not sensitive to the number of hidden units l .
For the parameter λ, increasing λ from 0.7 to 0.99 generally increases the Micro-F1 score on Cora, especially when the number of labelled nodes is small. In the meantime, λ = 0.99 still provides slight better performance than λ = 0.7 on Co-CS. To conclude, we should avoid relatively low λ, e.g., lower than 0.8, to prevent inferior performance when the labelling budget is small.

B.5 Experiment Environments
We conduct all the experiments on a hardware platform with Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz and a single GeForce RTX 2080 Ti GPU. We use the following software packages used for implementing LSCALE and other baselines: