Learning First-to-Spike Policies for Neuromorphic Control Using Policy
  Gradients
  Artificial Neural Networks (ANNs) are currently being used as function
approximators in many state-of-the-art Reinforcement Learning (RL) algorithms.
Spiking Neural Networks (SNNs) have been shown to drastically reduce the energy
consumption of ANNs by encoding information in sparse temporal binary spike
streams, hence emulating the communication mechanism of biological neurons. Due
to their low energy consumption, SNNs are considered to be important candidates
as co-processors to be implemented in mobile devices. In this work, the use of
SNNs as stochastic policies is explored under an energy-efficient
first-to-spike action rule, whereby the action taken by the RL agent is
determined by the occurrence of the first spike among the output neurons. A
policy gradient-based algorithm is derived considering a Generalized Linear
Model (GLM) for spiking neurons. Experimental results demonstrate the
capability of online trained SNNs as stochastic policies to gracefully trade
energy consumption, as measured by the number of spikes, and control
performance. Significant gains are shown as compared to the standard approach
of converting an offline trained ANN into an SNN.
