Training Feedback Spiking Neural Networks by Implicit Differentiation on
  the Equilibrium State
  Spiking neural networks (SNNs) are brain-inspired models that enable
energy-efficient implementation on neuromorphic hardware. However, the
supervised training of SNNs remains a hard problem due to the discontinuity of
the spiking neuron model. Most existing methods imitate the backpropagation
framework and feedforward architectures for artificial neural networks, and use
surrogate derivatives or compute gradients with respect to the spiking time to
deal with the problem. These approaches either accumulate approximation errors
or only propagate information limitedly through existing spikes, and usually
require information propagation along time steps with large memory costs and
biological implausibility. In this work, we consider feedback spiking neural
networks, which are more brain-like, and propose a novel training method that
does not rely on the exact reverse of the forward computation. First, we show
that the average firing rates of SNNs with feedback connections would gradually
evolve to an equilibrium state along time, which follows a fixed-point
equation. Then by viewing the forward computation of feedback SNNs as a
black-box solver for this equation, and leveraging the implicit differentiation
on the equation, we can compute the gradient for parameters without considering
the exact forward procedure. In this way, the forward and backward procedures
are decoupled and therefore the problem of non-differentiable spiking functions
is avoided. We also briefly discuss the biological plausibility of implicit
differentiation, which only requires computing another equilibrium. Extensive
experiments on MNIST, Fashion-MNIST, N-MNIST, CIFAR-10, and CIFAR-100
demonstrate the superior performance of our method for feedback models with
fewer neurons and parameters in a small number of time steps. Our code is
avaiable at https://github.com/pkuxmq/IDE-FSNN.
