Coloring black boxes: visualization of neural network decisions
  Neural networks are commonly regarded as black boxes performing
incomprehensible functions. For classification problems networks provide maps
from high dimensional feature space to K-dimensional image space. Images of
training vector are projected on polygon vertices, providing visualization of
network function. Such visualization may show the dynamics of learning, allow
for comparison of different networks, display training vectors around which
potential problems may arise, show differences due to regularization and
optimization procedures, investigate stability of network classification under
perturbation of original vectors, and place new data sample in relation to
training data, allowing for estimation of confidence in classification of a
given sample. An illustrative example for the three-class Wine data and
five-class Satimage data is described. The visualization method proposed here
is applicable to any black box system that provides continuous outputs.
