Node Feature Extraction by Self-Supervised Multi-scale Neighborhood
  Prediction
  Learning on graphs has attracted significant attention in the learning
community due to numerous real-world applications. In particular, graph neural
networks (GNNs), which take numerical node features and graph structure as
inputs, have been shown to achieve state-of-the-art performance on various
graph-related learning tasks. Recent works exploring the correlation between
numerical node features and graph structure via self-supervised learning have
paved the way for further performance improvements of GNNs. However, methods
used for extracting numerical node features from raw data are still
graph-agnostic within standard GNN pipelines. This practice is sub-optimal as
it prevents one from fully utilizing potential correlations between graph
topology and node attributes. To mitigate this issue, we propose a new
self-supervised learning framework, Graph Information Aided Node feature
exTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification
(XMC) formalism, which is crucial for fine-tuning the language model based on
graph information, and scales to large datasets. We also provide a theoretical
analysis that justifies the use of XMC over link prediction and motivates
integrating XR-Transformers, a powerful method for solving XMC problems, into
the GIANT framework. We demonstrate the superior performance of GIANT over the
standard GNN pipeline on Open Graph Benchmark datasets: For example, we improve
the accuracy of the top-ranked method GAMLP from $68.25\%$ to $69.67\%$, SGC
from $63.29\%$ to $66.10\%$ and MLP from $47.24\%$ to $61.10\%$ on the
ogbn-papers100M dataset by leveraging GIANT.
