Learning sparsity in reservoir computing through a novel bio-inspired
  algorithm
  The mushroom body is the key network for the representation of learned
olfactory stimuli in Drosophila and insects. The sparse activity of Kenyon
cells, the principal neurons in the mushroom body, plays a key role in the
learned classification of different odours. In the specific case of the fruit
fly, the sparseness of the network is enforced by an inhibitory feedback neuron
called APL, and by an intrinsic high firing threshold of the Kenyon cells. In
this work we took inspiration from the fruit fly brain to formulate a novel
machine learning algorithm that is able to optimize the sparsity level of a
reservoir by changing the firing thresholds of the nodes. The sparsity is only
applied on the readout layer so as not to change the timescales of the
reservoir and to allow the derivation of a one-layer update rule for the firing
thresholds. The proposed algorithm is a combination of learning a
neuron-specific sparsity threshold via gradient descent and a global sparsity
threshold via a Markov chain Monte Carlo method. The proposed model outperforms
the standard gradient descent, which is limited to the readout weights of the
reservoir, on two example tasks. It demonstrates how the learnt sparse
representation can lead to better classification performance, memorization
ability and convergence time.
