Mutual information estimation for graph convolutional neural networks
  Measuring model performance is a key issue for deep learning practitioners.
However, we often lack the ability to explain why a specific architecture
attains superior predictive accuracy for a given data set. Often, validation
accuracy is used as a performance heuristic quantifying how well a network
generalizes to unseen data, but it does not capture anything about the
information flow in the model. Mutual information can be used as a measure of
the quality of internal representations in deep learning models, and the
information plane may provide insights into whether the model exploits the
available information in the data. The information plane has previously been
explored for fully connected neural networks and convolutional architectures.
We present an architecture-agnostic method for tracking a network's internal
representations during training, which are then used to create the mutual
information plane. The method is exemplified for graph-based neural networks
fitted on citation data. We compare how the inductive bias introduced in
graph-based architectures changes the mutual information plane relative to a
fully connected neural network.
