Graph-adaptive Rectified Linear Unit for Graph Neural Networks
  Graph Neural Networks (GNNs) have achieved remarkable success by extending
traditional convolution to learning on non-Euclidean data. The key to the GNNs
is adopting the neural message-passing paradigm with two stages: aggregation
and update. The current design of GNNs considers the topology information in
the aggregation stage. However, in the updating stage, all nodes share the same
updating function. The identical updating function treats each node embedding
as i.i.d. random variables and thus ignores the implicit relationships between
neighborhoods, which limits the capacity of the GNNs. The updating function is
usually implemented with a linear transformation followed by a non-linear
activation function. To make the updating function topology-aware, we inject
the topological information into the non-linear activation function and propose
Graph-adaptive Rectified Linear Unit (GReLU), which is a new parametric
activation function incorporating the neighborhood information in a novel and
efficient way. The parameters of GReLU are obtained from a hyperfunction based
on both node features and the corresponding adjacent matrix. To reduce the risk
of overfitting and the computational cost, we decompose the hyperfunction as
two independent components for nodes and features respectively. We conduct
comprehensive experiments to show that our plug-and-play GReLU method is
efficient and effective given different GNN backbones and various downstream
tasks.
