Learning the Number of Neurons in Deep Networks
  Nowadays, the number of layers and of neurons in each layer of a deep network
are typically set manually. While very deep and wide networks have proven
effective in general, they come at a high memory and computation cost, thus
making them impractical for constrained platforms. These networks, however, are
known to have many redundant parameters, and could thus, in principle, be
replaced by more compact architectures. In this paper, we introduce an approach
to automatically determining the number of neurons in each layer of a deep
network during learning. To this end, we propose to make use of structured
sparsity during learning. More precisely, we use a group sparsity regularizer
on the parameters of the network, where each group is defined to act on a
single neuron. Starting from an overcomplete network, we show that our approach
can reduce the number of parameters by up to 80\% while retaining or even
improving the network accuracy.
