Implicit vs Unfolded Graph Neural Networks
  It has been observed that graph neural networks (GNN) sometimes struggle to
maintain a healthy balance between the efficient modeling long-range
dependencies across nodes while avoiding unintended consequences such
oversmoothed node representations or sensitivity to spurious edges. To address
this issue (among other things), two separate strategies have recently been
proposed, namely implicit and unfolded GNNs. The former treats node
representations as the fixed points of a deep equilibrium model that can
efficiently facilitate arbitrary implicit propagation across the graph with a
fixed memory footprint. In contrast, the latter involves treating graph
propagation as unfolded descent iterations as applied to some graph-regularized
energy function. While motivated differently, in this paper we carefully
quantify explicit situations where the solutions they produce are equivalent
and others where their properties sharply diverge. This includes the analysis
of convergence, representational capacity, and interpretability. In support of
this analysis, we also provide empirical head-to-head comparisons across
multiple synthetic and public real-world node classification benchmarks. These
results indicate that while IGNN is substantially more memory-efficient, UGNN
models support unique, integrated graph attention mechanisms and propagation
rules that can achieve SOTA node classification accuracy across disparate
regimes such as adversarially-perturbed graphs, graphs with heterophily, and
graphs involving long-range dependencies.
