Training Deep Spiking Auto-encoders without Bursting or Dying Neurons
  through Regularization
  Spiking neural networks are a promising approach towards next-generation
models of the brain in computational neuroscience. Moreover, compared to
classic artificial neural networks, they could serve as an energy-efficient
deployment of AI by enabling fast computation in specialized neuromorphic
hardware. However, training deep spiking neural networks, especially in an
unsupervised manner, is challenging and the performance of a spiking model is
significantly hindered by dead or bursting neurons. Here, we apply end-to-end
learning with membrane potential-based backpropagation to a spiking
convolutional auto-encoder with multiple trainable layers of leaky
integrate-and-fire neurons. We propose bio-inspired regularization methods to
control the spike density in latent representations. In the experiments, we
show that applying regularization on membrane potential and spiking output
successfully avoids both dead and bursting neurons and significantly decreases
the reconstruction error of the spiking auto-encoder. Training regularized
networks on the MNIST dataset yields image reconstruction quality comparable to
non-spiking baseline models (deterministic and variational auto-encoder) and
indicates improvement upon earlier approaches. Importantly, we show that,
unlike the variational auto-encoder, the spiking latent representations display
structure associated with the image class.
