SpikeGrad: An ANN-equivalent Computation Model for Implementing
  Backpropagation with Spikes
  Event-based neuromorphic systems promise to reduce the energy consumption of
deep learning tasks by replacing expensive floating point operations on dense
matrices by low power sparse and asynchronous operations on spike events. While
these systems can be trained increasingly well using approximations of the
back-propagation algorithm, these implementations usually require high
precision errors for training and are therefore incompatible with the typical
communication infrastructure of neuromorphic circuits. In this work, we analyze
how the gradient can be discretized into spike events when training a spiking
neural network. To accelerate our simulation, we show that using a special
implementation of the integrate-and-fire neuron allows us to describe the
accumulated activations and errors of the spiking neural network in terms of an
equivalent artificial neural network, allowing us to largely speed up training
compared to an explicit simulation of all spike events. This way we are able to
demonstrate that even for deep networks, the gradients can be discretized
sufficiently well with spikes if the gradient is properly rescaled. This form
of spike-based backpropagation enables us to achieve equivalent or better
accuracies on the MNIST and CIFAR10 dataset than comparable state-of-the-art
spiking neural networks trained with full precision gradients. The algorithm,
which we call SpikeGrad, is based on accumulation and comparison operations and
can naturally exploit sparsity in the gradient computation, which makes it an
interesting choice for a spiking neuromorphic systems with on-chip learning
capacities.
