Sampling the "Inverse Set" of a Neuron: An Approach to Understanding
  Neural Nets
  With the recent success of deep neural networks in computer vision, it is
important to understand the internal working of these networks. What does a
given neuron represent? The concepts captured by a neuron may be hard to
understand or express in simple terms. The approach we propose in this paper is
to characterize the region of input space that excites a given neuron to a
certain level; we call this the inverse set. This inverse set is a complicated
high dimensional object that we explore by an optimization-based sampling
approach. Inspection of samples of this set by a human can reveal regularities
that help to understand the neuron. This goes beyond approaches which were
limited to finding an image which maximally activates the neuron or using
Markov chain Monte Carlo to sample images, but this is very slow, generates
samples with little diversity and lacks control over the activation value of
the generated samples. Our approach also allows us to explore the intersection
of inverse sets of several neurons and other variations.
