Temporal Surrogate Back-propagation for Spiking Neural Networks
  Spiking neural networks (SNN) are usually more energy-efficient as compared
to Artificial neural networks (ANN), and the way they work has a great
similarity with our brain. Back-propagation (BP) has shown its strong power in
training ANN in recent years. However, since spike behavior is
non-differentiable, BP cannot be applied to SNN directly. Although prior works
demonstrated several ways to approximate the BP-gradient in both spatial and
temporal directions either through surrogate gradient or randomness, they
omitted the temporal dependency introduced by the reset mechanism between each
step. In this article, we target on theoretical completion and investigate the
effect of the missing term thoroughly. By adding the temporal dependency of the
reset mechanism, the new algorithm is more robust to learning-rate adjustments
on a toy dataset but does not show much improvement on larger learning tasks
like CIFAR-10. Empirically speaking, the benefits of the missing term are not
worth the additional computational overhead. In many cases, the missing term
can be ignored.
