Graph Structure of Neural Networks
  Neural networks are often represented as graphs of connections between
neurons. However, despite their wide use, there is currently little
understanding of the relationship between the graph structure of the neural
network and its predictive performance. Here we systematically investigate how
does the graph structure of neural networks affect their predictive
performance. To this end, we develop a novel graph-based representation of
neural networks called relational graph, where layers of neural network
computation correspond to rounds of message exchange along the graph structure.
Using this representation we show that: (1) a "sweet spot" of relational graphs
leads to neural networks with significantly improved predictive performance;
(2) neural network's performance is approximately a smooth function of the
clustering coefficient and average path length of its relational graph; (3) our
findings are consistent across many different tasks and datasets; (4) the sweet
spot can be identified efficiently; (5) top-performing neural networks have
graph structure surprisingly similar to those of real biological neural
networks. Our work opens new directions for the design of neural architectures
and the understanding on neural networks in general.
