Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation
Spiking Neural Network (SNN) is a promising energy-efficient AI model when
implemented on neuromorphic hardware. However, it is a challenge to efficiently
train SNNs due to their non-differentiability. Most existing methods either
suffer from high latency (i.e., long simulation time steps), or cannot achieve
as high performance as Artificial Neural Networks (ANNs). In this paper, we
propose the Differentiation on Spike Representation (DSR) method, which could
achieve high performance that is competitive to ANNs yet with low latency.
First, we encode the spike trains into spike representation using (weighted)
firing rate coding. Based on the spike representation, we systematically derive
that the spiking dynamics with common neural models can be represented as some
sub-differentiable mapping. With this viewpoint, our proposed DSR method trains
SNNs through gradients of the mapping and avoids the common
non-differentiability problem in SNN training. Then we analyze the error when
representing the specific mapping with the forward computation of the SNN. To
reduce such error, we propose to train the spike threshold in each layer, and
to introduce a new hyperparameter for the neural models. With these components,
the DSR method can achieve state-of-the-art SNN performance with low latency on
both static and neuromorphic datasets, including CIFAR-10, CIFAR-100, ImageNet,
and DVS-CIFAR10.