A new role for circuit expansion for learning in neural networks
  Many sensory pathways in the brain rely on sparsely active populations of
neurons downstream from the input stimuli. The biological reason for the
occurrence of expanded structure in the brain is unclear, but may be because
expansion can increase the expressive power of a neural network. In this work,
we show that expanding a neural network can improve its generalization
performance even in cases in which the expanded structure is pruned after the
learning period. To study this setting we use a teacher-student framework where
a perceptron teacher network generates labels which are corrupted with small
amounts of noise. We then train a student network that is structurally matched
to the teacher and can achieve optimal accuracy if given the teacher's synaptic
weights. We find that sparse expansion of the input of a student perceptron
network both increases its capacity and improves the generalization performance
of the network when learning a noisy rule from a teacher perceptron when these
expansions are pruned after learning. We find similar behavior when the
expanded units are stochastic and uncorrelated with the input and analyze this
network in the mean field limit. We show by solving the mean field equations
that the generalization error of the stochastic expanded student network
continues to drop as the size of the network increases. The improvement in
generalization performance occurs despite the increased complexity of the
student network relative to the teacher it is trying to learn. We show that
this effect is closely related to the addition of slack variables in artificial
neural networks and suggest possible implications for artificial and biological
neural networks.
