{'fraction_non_alphanumeric': 0.03923639317627945, 'fraction_numerical': 0.01050636339019767, 'mean_word_length': 4.8667196187450354, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Interpreting and Unifying Graph Neural Networks with An Optimization Framework

Graph Neural Networks (GNNs) have received considerable attention on graph-structured data learning for a wide variety of tasks. The well-designed propagation mechanism which has been demonstrated effective is the most fundamental part of GNNs. Although most of GNNs basically follow a message passing manner, litter effort has been made to discover and analyze their essential relations. In this paper, we establish a surprising connection between different propagation mechanisms with a unified optimization problem, showing that despite the proliferation of various GNNs, in fact, their proposed propagation mechanisms are the optimal solution optimizing a feature fitting function over a wide class of graph kernels with a graph regularization term. Our proposed unified optimization framework, summarizing the commonalities between several of the most representative GNNs, not only provides a macroscopic view on surveying the relations between different GNNs, but also further opens up new opportunities for flexibly designing new GNNs. With the proposed framework, we discover that existing works usually utilize naive graph convolutional kernels for feature fitting function, and we further develop two novel objective functions considering adjustable graph kernels showing low-pass or high-pass filtering capabilities respectively. Moreover, we provide the convergence proofs and expressive power comparisons for the proposed models. Extensive experiments on benchmark datasets clearly show that the proposed GNNs not only outperform the state-of-the-art methods but also have good ability to alleviate over-smoothing, and further verify the feasibility for designing GNNs with our unified optimization framework.


INTRODUCTION
Network is a ubiquitous structure for real-world data, such as social networks, citation networks and financial networks. Recently, Graph Neural Networks (GNNs) have gained great popularity in tackling the analytics tasks [7,14,31] on network-structured data. Moreover, GNNs have also been successfully applied to a wide range of application tasks, including recommendation [8,28], urban data mining [5,34], natural language processing [9,48] and computer vision [27,29].
The classical graph neural networks can be generally divided into two types: spectral-based GNNs and spatial-based GNNs. Spectralbased methods [6,37] mainly focus on defining spectral graph filters via graph convolution theorem; spatial-based methods [10,38] usually follow a message passing manner, where the most essential part is the feature propagation process along network topology. To date, many representative GNNs have been proposed by designing different feature propagation mechanisms, e.g., attention mechanism [30], personalized pagerank [15] and jump connection [39]. The well-designed propagation mechanism which has been demonstrated effective is the most fundamental part of GNNs. Although there are various propagation mechanisms, they basically utilize network topology and node features through aggregating node features along network topology. In view of this, one question naturally arises: Albeit with different propagation strategies, is there a unified mathematical guideline that essentially governs the propagation mechanisms of different GNNs? If so, what is it? A well informed answer to this question can provide a macroscopic view on surveying the relationships and differences between different GNNs in a principled way. Such mathematical guideline, once discovered, is able to help us identify the weakness of current GNNs, and further motivates more novel GNNs to be proposed.
As the first contribution of our work, we analyze the propagation process of several representative GNNs (e.g., GCN [14] and PPNP [15]), and abstract their commonalities. Surprisingly, we discover that they can be fundamentally summarized to a unified optimization framework with flexible graph convolutional kernels. The learned representation after propagation can be viewed as the optimal solution of the corresponding optimization objective implicitly. This unified framework consists of two terms: feature fitting term and graph Laplacian regularization term. The feature fitting term, building the relationship between node representation and original node features, is usually designed to meet different needs of specific GNNs. Graph Laplacian regularization term, playing the role of feature smoothing with topology, is shared by all these GNNs. For example, the propagation of GCN can be interpreted only by the graph Laplacian regularization term while PPNP needs another fitting term to constrain the similarity of the node representation and the original features.
Thanks to the macroscopic view on different GNNs provided by the proposed unified framework, the weakness of current GNNs is easy to be identified. As a consequence, the unified framework opens up new opportunities for designing novel GNNs. Traditionally, when we propose a new GNN model, we usually focus on designing specific spectral graph filter or aggregation strategy. Now, the unified framework provides another new path to achieve this, i.e., the new GNN can be derived by optimizing an objective function. In this way, we clearly know the optimization objective behind the propagation process, making the new GNN more interpretable and more reliable. Here, with the proposed framework, we discover that existing works usually utilize naïve graph convolutional kernels for feature fitting function, and then develop two novel flexible objective functions with adjustable kernels showing low-pass and high-pass filtering capabilities. We show that two corresponding graph neural networks with flexible graph convolutional kernels can be easily derived. Moreover, we also give the convergence ability analysis and expressive power comparisons for these two GNNs. The main contributions are summarized as follows: • We propose a unified objective optimization framework with a feature fitting function and a graph regularization term, and theoretically prove that this framework is able to unify a series of GNNs propagation mechanisms, providing a macroscopic perspective on understanding GNNs and bringing new insight for designing novel GNNs. • Within the proposed optimization framework, we design two novel deep GNNs with flexible low-frequency and highfrequency filters which can well alleviate over-smoothing. The theoretical analysis on both of their convergence and excellent expressive power is provided. • Our extensive experiments on series of benchmark datasets clearly show that the proposed two GNNs outperform the state-of-the-art methods. This further verifies the feasibility for designing GNNs under the unified framework.

RELATED WORK
Graph Neural Networks. The current graph neural networks can be broadly divided into two categories: spectral-based GNNs and spatial-based GNNs. Spectral-based GNNs define graph convolutional operations in Fourier domain by designing spectral graph filters. [3] generalizes CNNs to graph signal based on the spectrum of graph Laplacian. ChebNet [6] uses Chebyshev polynomials to approximate the -order localized graph filters. GCN [14] employs the 1-order simplification of the Chebyshev filter. GWNN [37] leverages sparse and localized graph wavelet transform to design spectral GNNs. Spatial-based GNNs directly design aggregation strategies along network topology, i.e., feature propagation mechanisms. GCN [14] directly aggregates one-hop neighbors along topology. GAT [30] utilizes attention mechanisms to adaptively learn aggregation weights. GraphSAGE [10] uses mean/max/LSTM pooling for aggregation. MixHop [1] aggregates neighbors at various distances to capture mixing relationships. GIN [38] uses a simple but expressive injective multiset function for neighbor aggregation. Policy-GNN [16] uses a meta-policy framework to adaptively learn the aggregation policy. Furthermore, there are some advanced topics have been studied in GNNs. For example, non-Euclidean space graph neural networks [2,42]; heterogeneous graph neural networks [33,44]; explanations for graph neural networks [41,43]; pre-training graph neural networks [12,25]; robust graph neural networks [13,47]. For more details, please find in [36,45,46] survey papers. Analysis and understanding on GNNs. Many works on understanding GNNs have been provided recently, which point out ways for designing and improving graph neural networks. Existing theoretical analysis works on GNNs are three-fold: 1) The spectral filtering characteristic analysis: Li et al. [17] show that the graph convolutional operation is a special form of Laplacian smoothing, and also point out the over-smoothing problem under many layers of graph convolutions; Wu et al. [35] make a simplification on GCN and theoretically analyze the resulting linear model acts as a fixed low-pass filter from spectral domain; NT et al. [21] also show that the graph convolutional operation is a simplified low-pass filter on original feature vectors and do not have the non-linear manifold learning property from the view of graph signal processing. 2) The over-smoothing problem analysis: Xu et al. [39] analyze the same over-smoothing problem by establishing the relationship between graph neural networks and random walk. And they show that GCN converges to the limit distribution of random walk as the number of layers increases; Chen et al. [4] provide theoretical analysis and imply that nodes with high degrees are more likely to suffer from over-smoothing in a multi-layer graph convolutional model. 3) The capability of GNNs analysis: Hou et al. [11] work on understanding how much performance GNNs actually gain from graph data and design two smoothness metrics to measure the quantity and quality of obtained information; Loukas et al. [19] show that GNNs are Turing universal under sufficient conditions on their depth, width and restricted depth and width may lose a significant portion of their power; Oono et al. [22] investigate the expressive power of GNNs as the layer size tends to infinity and show that deep GNNs can only preserve information of node degrees and connected components. However, these works do not theoretically analyze the intrinsic connections about the propagation mechanisms for GNNs.

A UNIFIED OPTIMIZATION FRAMEWORK
Notations. We consider graph convolutional operations on a graph G = (V, E) with node set V and edge set E, = |V | is the number of nodes. The nodes are described by the feature matrix X ∈ R × , where is the dimension of node feature. Graph structure of G can be described by the adjacency matrix A ∈ R × where A , = 1 if there is an edge between nodes and , otherwise 0. The diagonal degree matrix is denoted as D = ( 1 , · · · , ),

The Unified Framework
The well-designed propagation mechanisms of different GNNs basically follow similar propagating steps, i.e, node features aggregate and transform along network topology for a certain depth. Here, we summarize the -layer propagation mechanisms mainly as the following two forms.
For GNNs with layer-wise feature transformation (e.g. GCN [14]), the -layer propagation process can be represented as: with Z (0) = X and Z is the output representation after the -layer propagation. And , usually depending on specific GNN models, represents the generalized combination operation after convolutions. Agg G; Z ( −1) means aggregating the ( − 1)-layer output Z ( −1) along graph G for the -th convolutional operation, and Trans(·) is the corresponding layer-wise feature transformation operation including non-linear activation function () and layer-specific learnable weight matrix W.
Some deep graph neural networks (e.g. APPNP [15], DAGNN [18]) decouple the layer-wise Trans(·) and Agg G; Z ( −1) , and use a separated feature transformation before the consecutive aggregation steps: with Z (0) = Trans(X) and Trans(·) can be any linear or non-linear transformation operation on original feature matrix X.
In addition, the combination operation is generally two-fold: for GNNs like GCN, SGC, and APPNP, directly utilizes the -th layer output. And for GNNs using outputs from other layers, like JKNet and DAGNN, may represent pooling, concatenation or attention operations on the some (or all) outputs from layers.
Actually, the propagation process including aggregation and transformation is the key core of GNNs. Network topology and node features are the two most essential sources of information improving the learned representation during propagation: network topology usually plays the role of low-pass filter on the input node signals, which smooths the features of two connected nodes [17]. In this way, the learned node representation is able to capture the homophily of graph structure. As for the node feature, itself contains complex information, e.g., low-frequency and high-frequency information. Node feature can be flexibly used to further restrain the learned node representation. For example, APPNP adds the original node feature to the representation learned by each layer, which well preserves the personalized information so as to alleviate over-smoothing.
The above analysis implies that despite various GNNs are proposed with different propagation mechanisms, in fact, they usually potentially aim at achieving two goals: encoding useful information from feature and utilizing the smoothing ability of topology, which can be formally formulated as the following optimization objective: ( Here, is a non-negative coefficient, is usually chosen from [0, 1], and H is the transformation on original input feature matrix X. F 1 and F 2 are defined as arbitrary graph convolutional kernels. Z is the propagated representation and corresponds to the final propagation result when minimizing the objective O.
In this unified framework, the first part O is a fitting term which flexibly encodes the information in H to the learned representation Z through designing different graph convolutional kernels F 1 and F 2 . Graph convolutional kernels F 1 and F 2 can be chosen from the I,Â,L, showing the all-pass, low-pass, high-pass filtering capabilities respectively. The second term O is a graph Laplacian regularization term constraining the learned representations of two connected nodes become similar, so that the homophily property can be captured, and O comes from the following graph Laplacian regularization: In the following, we theoretically prove that the propagation mechanisms of some typical GNNs are actually the special cases of our proposed unified framework as shown in Table 1. This unified framework builds the connection among some typical GNNs, enabling us to interpret the current GNNs in a global perspective.
Simplifying Graph Convolutional Network (SGC) reduces this excess complexity through removing nonlinearities and collapsing weight matrices between consecutive layers. The linear model exhibits comparable performance since SGC has the similar propagation mechanism with GCN as: JKNet [39] Jumping to the last layer . We have the following interpretations on the propagation mode of SGC (GCN) under the proposed unified framework.
Theorem 3.1. With = 0 and = 1 in Eq. (3), the propagation process of SGC/GCN optimizes the following graph regularization term: where Z is initialized as XW*.
Proof. Set derivative of Eq. (7) with respect to Z to zero: Eq. (8) can be explained as an limit distribution where Z =ÂZ . Then we use the following iterative form to approximate the limit Z with → ∞: When SGC initializes input representation as Z (0) = XW*, Eq. (9) becomes: which matches the propagation mechanism of SGC. Since GCN can be simplified as SGC by ignoring the non-linear transformation, this conclusion also holds for GCN. □ Graph Convolutional Operation. The above analysis is for the consecutive layers graph convolutional operations, here, we also pay attention to the one layer graph convolutional operation (GC operation) with the following propagation mechanism: , the 1layer GC operation optimizes the following objective under first-order approximation: where H = XW is the linear transformation on feature, W is a trainable weight matrix.
Proof. Please refer to Appendix A.1 □

Interpreting PPNP and APPNP
PPNP [15] is a graph neural network which utilizes a propagation mechanism derived from personalized PageRank and separates the feature transformation from the aggregation process: where ∈ (0, 1] is the teleport probability, and H is the nonlinear transformation result of the original feature X using an MLP network (·).
Furthermore, due to the high complexity of calculating the inverse matrix, a power iterative version with linear computational complexity named APPNP is used for approximation. The propagation process of APPNP can be viewed as a layer-wise graph convolution with a residual connection to the initial transformed feature matrix H: Actually, it has been proved in [15] that APPNP converges to PPNP when → ∞, so we use one objective under the framework to explain both of them.
, the propagation process of PPNP/APPNP optimizes the following objective: where H = (X).
Proof. We can set the derivative of Eq. (15) with respect to Z to zero and get the optimal Z as: Note that a matrix M has an inverse matrix if and only if the determinant of matrix det(M) is not zero. Since the eigenvalues of the normalized Laplacian matrix ∈ [0, 2), and the eigenvalues of the matrix I +L are (1 + ) > 0. Then det(I +L) > 0 and Z in Eq. (16) can be rewritten as: We useÂ and to rewrite Eq. (17): which exactly corresponds to the propagation mechanism of PPNP or the convergence propagation result of APPNP. □

Interpreting JKNet and DAGNN
JKNet [39] is a deep graph neural network which exploits information from neighborhoods of differing locality. This architecture selectively combines aggregations from different layers with Concatenation/Max-pooling/Attention at the output, i.e., the representations "jump" to the last layer.
For convenience, following [4,35], we simplify the -th ( ∈ [1, ]) layer graph convolutional operation in the similar way by ignoring the non-linear activation with ( ) = and sharing W* = W (0) W (1) · · · W ( −1) for each layer. Then -th layer temporary output isÂ XW*. Using attention mechanism for combination at the last layer, the -layer propagation result of JKNet can be written as: Z = PROPAGATE(X; G; ) where 1 , 2 , ..., are the learnable fusion weights with =1 = 1, and for convenient analysis, we assume that all nodes of the -th layer share one common weight .
here H = XW* is the linear feature transformation after simplifications.
Proof. Similarly, we can set derivative of Eq. (20) with respect to Z to zero and get the optimal Z as: (21) Note that det(I +L) > 0, thus matrix {I +L} −1 exists. Then the corresponding closed-form solution can be written as: Since 1+ < 1 for ∀ > 0, and matrixÂ has absolute eigenvalues bounded by 1, thus, all its positive powers have bounded operator norm, then the inverse matrix can be decomposed as follows with → ∞: With H = XW*, we have the following expansion: Note that 1 1+ + (1+ ) 2 + · · · + −1 (1+ ) + · · · = 1 and we can change the coefficient ∈ (0, ∞) to fit fusion weights 1 , 2 , · · · , . When the layer is large enough, the propagation mechanism of JKNet in Eq. (19) approximately corresponds to the objective Eq. (20). □ DAGNN [18]. Deep Adaptive Graph Neural Networks (DAGNN) tries to adaptively incorporate information from large receptive fields. After decoupling representation transformation and propagation, the propagation mechanism of DAGNN is similar to that of JKNet: Z = PROPAGATE(X; G; ) H = (X) is the non-linear feature transformation using an MLP network, which is conducted before the propagation process, and 0 , 1 , · · · , are the learnable retainment scores where =0 = 1 and we assume that all nodes of the -th layer share one common weight for convenience.

Discussion
For clarity, we conclude the overall relations between different GNNs and the corresponding objective functions in Table 1. It can be seen that our proposed framework abstracts the commonalities between different promising representative GNNs. Based on the framework, we can understand their relationships much easier. For example, the corresponding optimization objective for SGC in Theorem 3.1 only has a graph regularization term, while the objective for APPNP in Theorem 3.3 has both fitting term and graph regularization term. The explicit difference of objective function well explains deep APPNP (PPNP) outperforms SGC (GCN) on over-smoothing problem by additionally requiring the learned representation to encode the original features.
On the other hand, our proposed framework shows a big picture of GNNs by mathematically modelling the objective optimization function. Considering that different existing GNNs can be fit into this framework, novel variations of GNNs can also be easily come up. All we need is to design the variables within this framework (e.g., different graph convolutional kernels F 1 and F 2 ) based on the specific scenarios, the corresponding propagation can be easily derived, and new GNNs architecture can be naturally designed. With one targeted objective function, the newly designed model is more interpretable and more reliable.

GNN-LF/HF: OUR PROPOSED MODELS
Based on the unified framework, we find that most of the current GNNs simply set F 1 and F 2 as I in feature fitting term, implying that they require all original information in H to be encoded into Z. However, in fact, the H may inevitably contain noise or uncertain information. We notice that JKNet has the propagation objective with F 2 asÂ, which can encode the low-frequency information in H to Z. While, in reality, the situation is more complex because it is hard to determine what information should be encoded, only considering one type of information cannot satisfy the needs of different downstream tasks, and sometimes high-frequency or all information is even also helpful. In this section, we focus on designing novel F 1 and F 2 to flexibly encode more comprehensive information under the framework.

GNN with Low-pass Filtering Kernel
where H = (X).
And we set ∈ (0, 2/3) to ensure the iterative approximation solution in subsection 4.1.3 has positive coefficients. By adjusting the balance coefficient , the designed objective can flexibly constrain the similarity of Z and H in both original and low-pass filtering spaces, which is beneficial to meet the needs of different tasks.

Closed Solution.
To minimize the objective function in Eq. (29), we set derivative of Eq. (29) with respect to Z to zero and derive the corresponding closed-form solution as follows: We can rewrite the Eq. (30) usingÂ as:

Iterative Approximation.
Considering that the closed-form solution is computationally inefficient because of the matrix inversion, we can use the following iterative approximation solution instead without constructing the dense inverse matrix: to propagate both H and AH, then we can get the representation encoding feature information from both original and low-frequency spaces.
GNN-LF-iter Using the iter-form propagation mechanism in Eq. (32), we can design a deep and computationally efficient graph neural network with ∈ [1/2, 1), ∈ (0, 2/3): (34) We directly use the -layer output as the propagation results. This iterative propagation mechanism can be viewed as layer-wiseÂ based neighborhood aggregation, with residual connection on feature matrix H and filtered feature matrixÂH. Note that we decouple the layer-wise transformation and aggregation process like [15,18], which is beneficial to alleviate the over-smoothing problem. GNN-LF-iter and GNN-LF-closed have the following relation: Proof. After the -layer propagation using GNN-LF-iter, the corresponding expansion result can be written as: where ∈ [1/2, 1), ∈ (0, 2/3) and | 1+ −2 1+ − | < 1. When → ∞, the left term tends to 0 and the right term becomes a geometric series. The series converges sinceÂ has absolute eigenvalues bounded by 1, then Eq. (35) can be rewritten as: which exactly is the equation for calculating GNN-LF-closed. □ The training of GNN-LF is also the same with other GNNs. For example, it evaluates the cross-entropy loss over all labeled examples for semi-supervised multi-class classification task.

GNN with High-pass Filtering Kernel
where H = (X).
Analogously, is also a balance coefficient, and we set ∈ (0, ∞) so that I +L = V * Λ Λ Λ * V * is a symmetric and positive semi-definite matrix and the matrix {I +L} 1/2 = V * Λ Λ Λ * 1/2 V * has a filtering behavior similar to that of {I +L}. As can be seen in Eq. (37), by adjusting the balance coefficient , the designed objectives can flexibly constrain the similarity of Z and H in both original and high-frequency spaces.

Closed Solution.
We calculate the closed-form solution as: it also can be rewritten as:

Iterative Approximation.
Considering it is inefficient to calculate the inverse matrix, we give the following iterative approximation solution without constructing the dense inverse matrix:

Model Design.
With the derived two propagation strategies in Eq. (39) and in Eq. (40), we propose two new GNNs in both closed and iterative forms. Similarly, we use GNN-HF to denote GNN with High-pass Filtering graph convolutional kernels. GNN-HF-closed Using the closed-form propagation matrix in Eq. (39), we define the following new graph neural networks with closed-form propagation mechanism: Note that ∈ (0, ∞) and ∈ (0, 1]. By applying the propagation matrix {( + 1/ )I + (1 − − 1/ )Â} −1 directly on both H and LH matrix, then we can get the representation encoding feature information from both original and high-frequency spaces. GNN-HF-iter Using the iterative propagation mechanism in Section 4.2.3, we have a deep and computationally efficient graph neural networks with ∈ (0, ∞) and ∈ (0, 1].
We directly use the -layer output as the propagation results. Similarly, this iterative propagation mechanism can be viewed as layer-wiseÂ based neighborhood aggregation, and residual connection on both feature matrix H and high-frequency filtered feature ma-trixLH. And we also decouple the layer-wise transformation and aggregation process during propagation. GNN-HF-iter and GNN-HF-closed have the following relation: Proof. Please refer to Appendix A.2. □

SPECTRAL EXPRESSIVE POWER ANALYSIS
In this section, we study several propagation mechanisms in graph spectral domain to examine their expressive power. A polynomial filter of order on a graph signal x is defined as =0L x, where is the corresponding polynomial coefficients. Similar with [4], we also assume that the graph signal x is non-negative and x can be converted into the input signal H under linear transformation. We aim to compare the polynomial coefficients for different GNNs and show that GNN-LF/HF with order flexible polynomial filter coefficients have better spectral expressive power. For concision, we mainly analyze the filter coefficients of GNN-LF, and the spectral analysis of SGC, PPNP, GNN-HF is in Appendix A.3.

Filter Coefficient Analysis
Analysis of GNN-LF. From the analysis in Theorem 4.2, we have the expanded propagation result of GNN-LF-iter in Eq. (35), which has been proved to converge to the propagation result of GNN-LF-closed with → ∞. Taking this propagation result for analysis, we have the following filtering expression when → ∞: (43) Expand the above equation, then the filter coefficients onL ( ∈ [0, ]) can be summarized into the following forms:

1) Filter coefficients forL
3) Filter coefficients forL : From the above analysis result on GNN-LF, we find the expression forms of filter coefficients depend on different and are determined by two adjustable factors and , which improve the expressive power of the spectral filters and further alleviate the over-smoothing problem. Analysis of SGC/PPNP/GNN-HF. Note that the analysis of GNN-HF is similar with that of GNN-LF, for concision, we show them in Appendix A.3.

Discussion on Expressive Power
As [18,39] point out, the reason for the over-smoothing problem is that typical GCN converges to the limit distribution of random walk which is isolated from the input feature and makes node representations inseparable as the number of layer increases. [4] also gives another understanding from the view of polynomial filtering coefficient and points out that flexible and arbitrary filter coefficients are essential for preventing over-smoothing. From the filter coefficients shown in Section 5.1 and Appendix A.3, we can find that: 1) SGC or -layer graph convolutional operations have fixed constant filtering coefficients, which limit the expressive power and further lead to over-smoothing. 2) PPNP has a better filtering expressive ability against SGC (GCN) since the filter coefficients of the order is changeable along with the factor . 3) Comparing with PPNP and SGC (GCN), GNN-LF/HF are more expressive under the influence of adjustable factors , or , which increase the ability to fit arbitrary coefficients of polynomial filter, and help GNN-LF/HF to alleviate the over-smoothing problem.
From the limit distributions of PPNP [15], GNN-LF in Eq. (36), GNN-HF in Eq. (53), we can also find that all of them converge to a distribution carrying information from both input feature and network structure. This property additionally helps to reduce the effects of over-smoothing on PPNP/GNN-LF/GNN-HF even if the number of layers goes to infinity.
Settings. We implement GNN-LF/HF based on Pytorch [24]. To ensure fair comparisons, we fix the hidden size as 64 for all models. We apply 2 regularization on the first layer parameter weights, with coefficients of 5e-3 on all datasets except 5e-4 for Wiki-CS. We set the learning rate = 0.01 for the other datasets except = 0.03 for Wiki-CS, and set dropout rate = 0.5. We Table 3: Node classification results (%). We show the average accuracy with uncertainties showing the 95% confidence level calculated by boot-strapping. Bold and underline are used to show the best and the runner-up results.
use the validation set for early stopping with a patience of 100 epochs. We fix 10 propagation depth for the two iterative version of GNN-LF/HF. Note that for APPNP and PPNP, all the settings are consistent with the above descriptions. As for ChebNet, GCN, GAT, SGC and GraphSAGE, we use the implementations of DGL [32] 1 . For JKNet and IncepGCN, we use the implementation in [26] 2 . We try to turn all hyperparameters reasonably to get the best performance, so some models even achieve better results than original reports. For JKNet, IncepGCN and SGC, we choose the best results of them with no more than 10 propagation depth. We conduct 10 runs on all datasets with the fixed training/validation/test split, where 20 nodes per class are used for training and 500/1000 nodes are used for val/test. For cora/citeseer/pubmed datasets, we follow the dataset splits in [40].

Node Classification
We evaluate the effectiveness of GNN-LF/HF against several stateof-the-art baselines on semi-supervised node classification task. We use accuracy (ACC) metric for evaluation, and report the average ACC with uncertainties showing the 95% confidence level calculated by bootstrapping in Table 3. We have the following observations: 1) GNN-LF and GNN-HF consistently outperform all the stateof-the-art baselines on all datasets. The best and the runner-up results are always achieved by GNN-LF/HF, which demonstrates the effectiveness of our proposed model. From the perspective of the unified objective framework, it is easy to check that GNN-LF/HF not only keep the representation same with the original features, but also consider capturing their similarities based on low-frequency or high-frequency information. These two relations are balanced so as to extract more meaningful signals and thus perform better.
2) From the results of the closed and iterative versions of GNN-LF/HF, we can see that using 10 propagation depth for GNN-LFiter/GNN-HF-iter is able to effectively approximate the GNN-LFclosed/GNN-HF-closed. As for performance comparisons between GNN-LF and GNN-HF, we find that it is hard to determine which is the best, since which filter works better may depend on the characteristic of different datasets. But in summary, flexibly and comprehensively considering multiple information in a GNN model can always achieve satisfactory results on different networks.
3) In addition, PPNP/APPNP always perform better than GCN/SGC since their objective also considers a fitting term to help find important information from features during propagation. On the other hand, APPNP outperforms JKNet mainly because that its propagation process takes full advantage of the original features and APPNP even decouples the layer-wise non-linear transformation operations [18] without suffering from performance degradation. Actually, the above differences of models and explanations for results can be easily drawn from our unified framework. datasets. For all the models, we set the hidden size as 64 and tune hyperparameters reasonably to get the best performance. Note that because of the specific architecture of JKNet and IncepGCN in [26], they are analyzed from 3 layers and IncepGCN on pubmed dataset faces the out of memory problem when the depth is bigger than 34. Figure 1 shows the accuracy with different propagation depths, and we have the following observations:

Propagation Depth Analysis
GNN-LF/HF and APPNP greatly alleviate the over-smoothing problem, since the performance does not drop when the depth increases. Furthermore, GNN-LF/HF are more flexible and more expressive with higher results under the influence of extra graph filters and three adjustable impact factors , and . As analyzed before, the polynomial filter coefficients of GNN-LF/HF are further more expressive and flexible than APPNP, GCN or SGC, which is useful for mitigating over-smoothing problem. Accuracy breaks down seriously for GCN while it drops a little bit slowly for SGC, but both GCN and SGC face the over-smoothing problem since the fixed polynomial filter coefficients limit their expressive power. As for JKNet/IncepGCN, they are deep GNNs to alleviate oversmoothing problem but still have to face performance degradation when the propagation depth increases.

Model Analysis
In this section, we analyze the performance of GNN-LF/HF with different impact factors: teleport probability , balance coefficient and balance coefficient . In general, adjusts the regularization term weight and has an effect on structural information during propagation; and focus on adjusting the balance weights between different filters and have effects on feature information during propagation.
We carefully adjust the value of three impact factors with GNN-LF-closed/GNN-HF-closed models on cora and citeseer datasets, and draw the contour map for accuracy in Figure 2. As can be seen: 1) For GNN-LF, plays a more dominant influence than . The classification accuracy exactly increases as becomes larger, and with the continuous increase of , the results begin to drop. Generally, the best performance can be achieved when ∈ [1 − 2, 5 − 1]. On the other hand, the accuracy is relatively stable with different , and generally speaking, is with a suitable range around [0.6, 0.9]. 2) For GNN-HF, and both play dominant influence, the suitable weight range for is also [1e-2, 5e-1] and lager may result in performance degradation. In general, our proposed GNN-LF/HF achieve stable and excellent performance within a wide changing range of these impact factors and or .
We then analyze the influence of balance coefficients and with the fixed on pubmed dataset, shown in Figure 3, and we have similar conclusions: For GNN-LF, the performance first increases stably as grows and then may show a slight drop after a certain threshold. The appropriate range for best performance is [0.6, 0.9]. For GNN-LF, the classification accuracy first increases and then drops with the rise of after a certain threshold.

CONCLUSION
The intrinsic relation for the propagation mechanisms of different GNNs is studied in this paper. We establish the connection between different GNNs and a flexible objective optimization framework. The proposed unified framework provides a global view on understanding and analyzing different GNNs, which further enables us to identify the weakness of current GNNs. Then we propose two novel GNNs with adjustable convolutional kernels showing low-pass and high-pass filtering capabilities, and their excellent expressive power is analyzed as well. Extensive experiments well demonstrate the superior performance of these two GNNs over the state-of-the-art models on real world datasets.