Towards Understanding the Effect of Leak in Spiking Neural Networks
  Spiking Neural Networks (SNNs) are being explored to emulate the astounding
capabilities of human brain that can learn and compute functions robustly and
efficiently with noisy spiking activities. A variety of spiking neuron models
have been proposed to resemble biological neuronal functionalities. With
varying levels of bio-fidelity, these models often contain a leak path in their
internal states, called membrane potentials. While the leaky models have been
argued as more bioplausible, a comparative analysis between models with and
without leak from a purely computational point of view demands attention. In
this paper, we investigate the questions regarding the justification of leak
and the pros and cons of using leaky behavior. Our experimental results reveal
that leaky neuron model provides improved robustness and better generalization
compared to models with no leak. However, leak decreases the sparsity of
computation contrary to the common notion. Through a frequency domain analysis,
we demonstrate the effect of leak in eliminating the high-frequency components
from the input, thus enabling SNNs to be more robust against noisy
spike-inputs.
