Emergence of Network Motifs in Deep Neural Networks
  Network science can offer fundamental insights into the structural and
functional properties of complex systems. For example, it is widely known that
neuronal circuits tend to organize into basic functional topological modules,
called "network motifs". In this article we show that network science tools can
be successfully applied also to the study of artificial neural networks
operating according to self-organizing (learning) principles. In particular, we
study the emergence of network motifs in multi-layer perceptrons, whose initial
connectivity is defined as a stack of fully-connected, bipartite graphs. Our
simulations show that the final network topology is primarily shaped by
learning dynamics, but can be strongly biased by choosing appropriate weight
initialization schemes. Overall, our results suggest that non-trivial
initialization strategies can make learning more effective by promoting the
development of useful network motifs, which are often surprisingly consistent
with those observed in general transduction networks.
