Investigating Expressiveness of Transformer in Spectral Domain for
  Graphs
  Transformers have been proven to be inadequate for graph representation
learning. To understand this inadequacy, there is need to investigate if
spectral analysis of transformer will reveal insights on its expressive power.
Similar studies already established that spectral analysis of Graph neural
networks (GNNs) provides extra perspectives on their expressiveness. In this
work, we systematically study and prove the link between the spatial and
spectral domain in the realm of the transformer. We further provide a
theoretical analysis that the spatial attention mechanism in the transformer
cannot effectively capture the desired frequency response, thus, inherently
limiting its expressiveness in spectral space. Therefore, we propose FeTA, a
framework that aims to perform attention over the entire graph spectrum
analogous to the attention in spatial space. Empirical results suggest that
FeTA provides homogeneous performance gain against vanilla transformer across
all tasks on standard benchmarks and can easily be extended to GNN based models
with low-pass characteristics (e.g., GAT). Furthermore, replacing the vanilla
transformer model with FeTA in recently proposed position encoding schemes has
resulted in comparable or better performance than transformer and GNN
baselines.
