Coloring the Black Box: Visualizing neural network behavior with a
  self-introspective model
  The following work presents how autoencoding all the possible hidden
activations of a network for a given problem can provide insight about its
structure, behavior, and vulnerabilities. The method, termed
self-introspection, can show that a trained model showcases similar activation
patterns (albeit randomly distributed due to initialization) when shown data
belonging to the same category, and classification errors occur in fringe areas
where the activations are not as clearly defined, suggesting some form of
random, slowly varying, implicit encoding occurring within deep networks, that
can be observed with this representation. Additionally, obtaining a
low-dimensional representation of all the activations allows for (1) real-time
model evaluation in the context of a multiclass classification problem, (2) the
rearrangement of all hidden layers by their relevance in obtaining a specific
output, and (3) the obtainment of a framework where studying possible
counter-measures to noise and adversarial attacks is possible.
Self-introspection can show how damaged input data can modify the hidden
activations, producing an erroneous response. A few illustrative are
implemented for feedforward and convolutional models and the MNIST and CIFAR-10
datasets, showcasing its capabilities as a model evaluation framework.
