Heterformer: A Transformer Architecture for Node Representation Learning on Heterogeneous Text-Rich Networks
We study node representation learning on heterogeneous text-rich networks,
where nodes and edges are multi-typed and some types of nodes are associated
with text information. Although recent studies on graph neural networks (GNNs)
and pretrained language models (PLMs) have demonstrated their power in encoding
network and text signals, respectively, less focus has been given to delicately
coupling these two types of models on heterogeneous text-rich networks.
Specifically, existing GNNs rarely model text in each node in a contextualized
way; existing PLMs can hardly be applied to characterize graph structures due
to their sequence architecture. In this paper, we propose Heterformer, a
Heterogeneous GNN-nested transformer that blends GNNs and PLMs into a unified
model. Different from previous "cascaded architectures" that directly add GNN
layers upon a PLM, our Heterformer alternately stacks two modules - a
graph-attention-based neighbor aggregation module and a transformer-based text
and neighbor joint encoding module - to facilitate thorough mutual enhancement
between network and text signals. Meanwhile, Heterformer is capable of
characterizing network heterogeneity and nodes without text information.
Comprehensive experiments on three large-scale datasets from different domains
demonstrate the superiority of Heterformer over state-of-the-art baselines in
link prediction, transductive/inductive node classification, node clustering,
and semantics-based retrieval.