Convolutional Neural Network Dynamics: A Graph Perspective
  The success of neural networks (NNs) in a wide range of applications has led
to increased interest in understanding the underlying learning dynamics of
these models. In this paper, we go beyond mere descriptions of the learning
dynamics by taking a graph perspective and investigating the relationship
between the graph structure of NNs and their performance. Specifically, we
propose (1) representing the neural network learning process as a time-evolving
graph (i.e., a series of static graph snapshots over epochs), (2) capturing the
structural changes of the NN during the training phase in a simple temporal
summary, and (3) leveraging the structural summary to predict the accuracy of
the underlying NN in a classification or regression task. For the dynamic graph
representation of NNs, we explore structural representations for
fully-connected and convolutional layers, which are key components of powerful
NN models. Our analysis shows that a simple summary of graph statistics, such
as weighted degree and eigenvector centrality, over just a few epochs can be
used to accurately predict the performance of NNs. For example, a weighted
degree-based summary of the time-evolving graph that is constructed based on 5
training epochs of the LeNet architecture achieves classification accuracy of
over 93%. Our findings are consistent for different NN architectures, including
LeNet, VGG, AlexNet and ResNet.
