Optimizing Sparse Matrix Multiplications for Graph Neural Networks
  Graph neural networks (GNNs) are emerging as a powerful technique for
modeling graph structures. Due to the sparsity of real-world graph data, GNN
performance is limited by extensive sparse matrix multiplication (SpMM)
operations involved in computation. While the right sparse matrix storage
format varies across input data, existing deep learning frameworks employ a
single, static storage format, leaving much room for improvement. This paper
investigates how the choice of sparse matrix storage formats affect the GNN
performance. We observe that choosing a suitable sparse matrix storage format
can significantly improve the GNN training performance, but the right format
depends on the input workloads and can change as the GNN iterates over the
input graph. We then develop a predictive model to dynamically choose a sparse
matrix storage format to be used by a GNN layer based on the input matrices.
Our model is first trained offline using training matrix samples, and the
trained model can be applied to any input matrix and GNN kernels with SpMM
computation. We implement our approach on top of PyTorch and apply it to 5
representative GNN models running on a multi-core CPU using real-life and
synthetic datasets. Experimental results show that our approach gives an
average speedup of 1.17x (up to 3x) for GNN running time.
