Transformer for Graphs: An Overview from Architecture Perspective
  Recently, Transformer model, which has achieved great success in many
artificial intelligence fields, has demonstrated its great potential in
modeling graph-structured data. Till now, a great variety of Transformers has
been proposed to adapt to the graph-structured data. However, a comprehensive
literature review and systematical evaluation of these Transformer variants for
graphs are still unavailable. It's imperative to sort out the existing
Transformer models for graphs and systematically investigate their
effectiveness on various graph tasks. In this survey, we provide a
comprehensive review of various Graph Transformer models from the architectural
design perspective. We first disassemble the existing models and conclude three
typical ways to incorporate the graph information into the vanilla Transformer:
1) GNNs as Auxiliary Modules, 2) Improved Positional Embedding from Graphs, and
3) Improved Attention Matrix from Graphs. Furthermore, we implement the
representative components in three groups and conduct a comprehensive
comparison on various kinds of famous graph data benchmarks to investigate the
real performance gain of each component. Our experiments confirm the benefits
of current graph-specific modules on Transformer and reveal their advantages on
different kinds of graph tasks.
