TopoAct: Visually Exploring the Shape of Activations in Deep Learning
  Deep neural networks such as GoogLeNet, ResNet, and BERT have achieved
impressive performance in tasks such as image and text classification. To
understand how such performance is achieved, we probe a trained deep neural
network by studying neuron activations, i.e., combinations of neuron firings,
at various layers of the network in response to a particular input. With a
large number of inputs, we aim to obtain a global view of what neurons detect
by studying their activations. In particular, we develop visualizations that
show the shape of the activation space, the organizational principle behind
neuron activations, and the relationships of these activations within a layer.
Applying tools from topological data analysis, we present TopoAct, a visual
exploration system to study topological summaries of activation vectors. We
present exploration scenarios using TopoAct that provide valuable insights into
learned representations of neural networks. We expect TopoAct to give a
topological perspective that enriches the current toolbox of neural network
analysis, and to provide a basis for network architecture diagnosis and data
anomaly detection.
