Sparse Activity and Sparse Connectivity in Supervised Learning
  Sparseness is a useful regularizer for learning in a wide range of
applications, in particular in neural networks. This paper proposes a model
targeted at classification tasks, where sparse activity and sparse connectivity
are used to enhance classification capabilities. The tool for achieving this is
a sparseness-enforcing projection operator which finds the closest vector with
a pre-defined sparseness for any given vector. In the theoretical part of this
paper, a comprehensive theory for such a projection is developed. In
conclusion, it is shown that the projection is differentiable almost everywhere
and can thus be implemented as a smooth neuronal transfer function. The entire
model can hence be tuned end-to-end using gradient-based methods. Experiments
on the MNIST database of handwritten digits show that classification
performance can be boosted by sparse activity or sparse connectivity. With a
combination of both, performance can be significantly better compared to
classical non-sparse approaches.
