Efficient Training of Spiking Neural Networks with Temporally-Truncated
  Local Backpropagation through Time
  Directly training spiking neural networks (SNNs) has remained challenging due
to complex neural dynamics and intrinsic non-differentiability in firing
functions. The well-known backpropagation through time (BPTT) algorithm
proposed to train SNNs suffers from large memory footprint and prohibits
backward and update unlocking, making it impossible to exploit the potential of
locally-supervised training methods. This work proposes an efficient and direct
training algorithm for SNNs that integrates a locally-supervised training
method with a temporally-truncated BPTT algorithm. The proposed algorithm
explores both temporal and spatial locality in BPTT and contributes to
significant reduction in computational cost including GPU memory utilization,
main memory access and arithmetic operations. We thoroughly explore the design
space concerning temporal truncation length and local training block size and
benchmark their impact on classification accuracy of different networks running
different types of tasks. The results reveal that temporal truncation has a
negative effect on the accuracy of classifying frame-based datasets, but leads
to improvement in accuracy on dynamic-vision-sensor (DVS) recorded datasets. In
spite of resulting information loss, local training is capable of alleviating
overfitting. The combined effect of temporal truncation and local training can
lead to the slowdown of accuracy drop and even improvement in accuracy. In
addition, training deep SNNs models such as AlexNet classifying CIFAR10-DVS
dataset leads to 7.26% increase in accuracy, 89.94% reduction in GPU memory,
10.79% reduction in memory access, and 99.64% reduction in MAC operations
compared to the standard end-to-end BPTT.
