Categorical Perception: A Groundwork for Deep Learning
  A well-known perceptual consequence of categorization in humans and other
animals, called categorical perception, is notably characterized by a
within-category compression and a between-category separation: two items, close
in input space, are perceived closer if they belong to the same category than
if they belong to different categories. Elaborating on experimental and
theoretical results in cognitive science, here we study categorical effects in
artificial neural networks. We combine a theoretical analysis that makes use of
mutual and Fisher information quantities, and a series of numerical simulations
on networks of increasing complexity. These formal and numerical analyses
provide insights into the geometry of the neural representation in deep layers,
with expansion of space near category boundaries and contraction far from
category boundaries. We investigate categorical representation by using two
complementary approaches: one mimics experiments in psychophysics and cognitive
neuroscience by means of morphed continua between stimuli of different
categories, while the other introduces a categoricality index that, for each
layer in the network, quantifies the separability of the categories at the
neural population level. We show on both shallow and deep neural networks that
category learning automatically induces categorical perception. We further show
that the deeper a layer, the stronger the categorical effects. As an outcome of
our study, we propose a coherent view of the efficacy of different heuristic
practices of the dropout regularization technique. More generally, our view,
which finds echoes in the neuroscience literature, insists on the differential
impact of noise in any given layer depending on the geometry of the neural
representation that is being learned, i.e. on how this geometry reflects the
structure of the categories.
