{'fraction_non_alphanumeric': 0.02516578813127019, 'fraction_numerical': 0.007623420053278921, 'mean_word_length': 4.726549821486531, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Supervised Learning in Temporally-Coded Spiking Neural Networks with Approximate Backpropagation

In this work we propose a new supervised learning method for temporally-encoded multilayer spiking networks to perform classification. The method employs a reinforcement signal that mimics backpropagation but is far less computationally intensive. The weight update calculation at each layer requires only local data apart from this signal. We also employ a rule capable of producing specific output spike trains; by setting the target spike time equal to the actual spike time with a slight negative offset for key high-value neurons the actual spike time becomes as early as possible. In simulated MNIST handwritten digit classification, two-layer networks trained with this rule matched the performance of a comparable backpropagation based non-spiking network.


I. INTRODUCTION
Backpropagation is a commonly used optimization procedure for Artificial Neural Networks (ANNs) in data classification, and has demonstrated particular success in recent years as increased processing speed has become available through massively parallel computing architectures. Despite this, the energy requirements of computing as applied to traditional ANNs continues to rise with the complexity of input data, limiting the versatility of machine learning in realworld applications. Spiking Neural Networks (SNNs), which incorporate the actual firing times, or spikes, of simulated neurons as part of their operational model, have been identified as a means to provide more energy efficient, event-based computing, while still theoretically still being capable of greatly increased computational power over ANNs [1]. Despite this, applying backpropagation to SNNs has proven to be challenging, since the spikes emitted by neurons have no smooth functional dependence on network parameters, making reliable gradient estimation an issue. Over the last decade there have been numerous attempts to address this spikegradient challenge, using various approximations to producing gradients that can be differentiated with respect to network parameters.
One of the first examples in combining backpropagation with an SNN is described in [2], where a feedforward network containing a single hidden layer was trained to associate spatiotemporal input patterns with desired network responses. The network's error-function was defined in terms of the squared difference between desired and actual firing times in the output layer, upon which gradient descent was performed to obtain weight gradients in each layer. To solve for the gradient of a discontinuous firing time, the authors assumed a linear dependence of this quantity on its activating input for a sufficiently small temporal region. While this approximation displayed success on limited, small-sized datasets, some drawbacks were noted such as the rule's reliance on small learning rates to ensure eventual convergence, as well as its limitation to just learning single-spike responses in each layer.
A later approach to combining backpropagation with SNNs has been proposed in [3] for learning desired input-output spike pattern associations. A first step taken by the authors of this study was to relate a previous spike-based supervised learning method, called ReSuMe [4], to an error-minimization procedure based on gradient descent. Secondly, the activations of neurons in a multilayer network were formulated in terms of their instantaneous firing rates, which were assumed to have a linear functional dependence on their previous layer inputs. Hence, by applying backpropagation to this approximatelyequivalent rate-based network, and then substituting neuronal firing rates for their actual spike trains, ReSuMe could be applied to multilayer learning. As a proof-of-concept the authors demonstrated the success of this rule on a small, benchmark data classification task. The main limitation of this approach relates to the assumption that neurons in the network can be treated as linear activations, which potentially diminishes the ability of the network to solve linearly nonseparable classification tasks.
A further example of backpropagation in SNNs was shown in [5], which extended a maximum-likelihood learning scheme from single to multilayer network structures. Building on the work of [6], this study considered networks containing stochastic spiking neurons, such that spike-gradients could be approximated by their expectations of generating sequences of output spikes. This probabilistic approach has the advantage of still retaining the non-linear behavior of the network, since the actual firing times of hidden neurons are taken into account when computing weight gradients. The capability of this rule was demonstrated quite extensively on a variety of associative learning tasks, although its technical performance on realworld datasets has not been tested.
A more recent examination of backpropagation in SNNs has taken advantage of the analytical tractability of nonleaky integrate-and-fire neurons constrained to single-spike responses [7]; Specifically, the author arrived at a closedform solution for backpropagation in a multi-layered network of such neurons, making it more easily extendable to SNNs containing more than one hidden layer. Furthermore, since just the first output spikes emitted by neurons were considered, the network was capable of classifying MNIST digits with minimal delay, with low test errors of around 3 %. The main limitation of this study relates to its restriction to just single spikes per neuron in all layers.
One of the most recent approaches to formulating spikebased backpropagation has relied on the use of 'surrogate gradients' in order to establish smooth functional derivatives with respect to network parameters [8]. Termed SuperSpike, this learning rule works to minimize a spike train (dis)similarity measure called the van Rossum Distance (vRD) [9] which is a function of the distance between target and actual output spike trains in the final layer of a multilayer SNN. In estimating the gradient of a spike train, the author used the slope of a sigmoid as an auxiliary function. In terms of the rule's performance, good accuracy was demonstrated in terms of its ability to precisely match arbitrary spatiotemporal spike patterns. As is the limitation with most spike-based backpropagation rules, SuperSpike relies on the presence of hidden spikes in order to allow the computation of weight gradients.
The above studies highlight the typical approach that is taken to training multilayer SNNs: selecting an appropriate output objective function, and establishing a smooth functional dependence with respect to the network's parameters such that backpropagation can be applied. Although this procedure has displayed success on certain learning tasks, there remains the issue that computing weight gradients in such a way is usually computationally expensive, and in most cases difficult to scale when applied to deeper network architectures. Certain model simplifications such as those described in the work of [7] help mitigate these issues, although this then trades off the increased computational capability afforded by SNNs when compared against ANNs.
Generally-speaking, training SNNs using an unsupervised learning method such as Spike-Timing-Dependent Plasticity (STDP) [10] or a single-layer, supervised learning rule [4], [6], [11]- [14] is inherently advantageous in terms of computational efficiency over spike-based backpropagation when computing weight gradients. This follows from their reliance on immediately available learning factors: specifically pre-and postsynaptic activity variables, whereas backpropagation also explicitly depends on an error signal that requires a series of backpropagation steps to be computed, starting from the final layer. Furthermore, as described previously, backpropagation applied to SNNs also relies extensively on approximate spikegradients, which for the most part is avoided with localized learning methods. Despite this, it is clear that backpropagation is better suited to training multilayer networks as applied to data classification tasks. This is apparent when considering that individualised error signals are backpropagated through the network, informing each synapse of its specific contribution to the network's overall accuracy during training. Hence, it becomes reasonable to suppose that a hybrid approach to SNN training combining localized layer-wise learning with generalised feedback signalling is capable of network performance approaching that of backpropagation, but without the sacrifice in computational cost.
In this paper we propose a novel hybrid learning algorithm for multilayer SNNs that uses one learning process for all of its layers. The algorithm builds on the work of [5], [14], beginning with a supervised learning method that circumvents the discontinuous spike-gradient issue by taking a maximumlikelihood approach to learning, which is then taken in the limit of a deterministic system to allow for more precise network responses [14]. With respect to each layer, the algorithm computes weight gradients by combining a set of localized learning factors with a simple reinforcement signal that carries a summary of the behavior of downstream layers in the network. This signal is condensed into a set of analog values associated with each postsynaptic neuron. This allows the learning algorithm to make decisions while armed with some knowledge of the global impact of each synapse without needing to backpropagate a complex error gradient function.
The rest of this paper is organized as follows. In Section II we describe the mathematical model of the neurons and input encoding, as well as the basic network structures employed. The learning algorithm is explained in Section III, and the simulation results given in Section IV. We conclude with a discussion in Section V.

II. NETWORK MODEL A. Spiking Neuron Model
This work uses the simplified Spike Response Model (SRM 0 ) to describe the dynamics of simulated neurons [10]. Specifically, a postsynaptic neuron's membrane potential u i at time t is described by . . }, is the spatiotemporal spike pattern of all presynaptic neurons indexed by j, and y i = {t i,1 , t i,2 , . . . } is the sequence of emitted spikes, or spike train, of the postsynaptic neuron with index i. On the righthand side of the equation, w ij is the synaptic weight between pre-and postsynaptic neurons j and i, respectively, describes the form of the Postsynaptic Potential (PSP) evoked at the postsynaptic neuron in response to a single presynaptic spike and κ is the reset kernel. The first double sum determines the net weighted PSP response due to all incoming spikes, while the second sum describes postsynaptic refractory effects due to the emission of output spikes. The kernel is given by where τ m = 10 ms and τ s = 5 ms are the membrane and synaptic time constants, respectively, and the coefficient 0 = 4 mV. Θ is the Heaviside step function. The reset kernel is given by where u r = 0 mV and V t = 15 mV are the reset and firing threshold potentials, respectively. In terms of spike-generation, the postsynaptic neuron fires a spike when its membrane potential exceeds V t , immediately after which its potential is reset to u r in response to Eq (3).

B. Network Structure
We will employ two different network structures in the experiments carried out in this paper. The first consists of a single SRM 0 neuron with either one or two input channels, each of which contributes a single spike at a preset time. This layout will be used to study the firing responses of individual neurons subject to the training scheme that will be described in Section III. The second network will be designed to test the performance of the proposed hybrid learning algorithm as applied to the MNIST handwritten digits classification problem, consisting of an input layer, at least one hidden layer and an output layer. The input layer fulfills the role of encoding digits as precisely-timed spikes, which shall be described in more detail in the following subsection. The hidden and output layers consist of SRM 0 neurons that perform computations on these inputs, which shall be trained using the rule proposed below. The hidden layers shall contain 100, 200 or 300 neurons, while the size of the output layer shall be held fixed at 10 neurons: each of which corresponds to one of the 10 handwritten digit classes. Whichever output neuron produces the first spike in response to a presented digit is used to decide the network's classification of the input. If no output neurons spike, no classification is made. The network is trained in 500 batches of 20 iterations each, as limited by available computing resources. Both the single-layer and multi-layer network layouts are simulated for 10 ms per iteration.

C. Temporal Encoding of Input
To solve the MNIST classification problem [15] we must first translate the images into input spikes. Specifically, the input layer, or layer 0, is set up to contain 784 channels, corresponding to the 784 pixels in the 28x28 image with oneto-one association. Each channel is then driven to produce up to one spike, where the spike timing depends on its associated pixel intensity. The timings of spikes are determined as follows. First, pixels are normalized such that their values fall within the range [0, 1]. We then choose a Gaussian sensitivity function for each timing, inspired by the population encoding model in [16]: where t 0 j (p j ) is the time of the input spike produced by channel j in the 0th layer as a function of the pixel strength p j , T = 10 ms is the duration of the simulated time and σ = 0.5 determines the breadth of the sensitivity function. Any pixel value below the threshold p t = 0.5 is considered not to produce a spike, as represented by an infinite response time. For a pixel value greater than the threshold, as the value increases the spike timing occurs closer to t 0 j (p j ) = 0, while a decrease in its value delays the spike up to a maximum delay of almost 4 ms. From a biological perspective, this rapid encoding scheme is supported by observations of neural populations in the brain being capable of encoding visual information using spikes occurring within a very small time window, on the order of around 10 ms [17].

A. FILTered-error (FILT) Learning Rule
We start our analysis by first describing the FILT synaptic plasticity rule, a supervised method which was originally derived in [14] to impose a target output spike train on an SRM 0 neuron. The distinction of this rule as compared with most other single-layer, spike-based learning rules is that, in the initial stage of its derivation, a stochastic firing rate substitution was used to circumvent the non-differentiable spike-gradient problem. Thereafter, in taking the limit of a deterministic spiking neuron with a fixed firing threshold, the following λ learning window, as a function of the separation between target and actual output spike times, was derived: where τ q = 10 ms, C m = τm τm+τq and C s = τs τs+τq . The τ q term is a spike-linkage variable, describing the coincidence between target and actual output spike times. Hence, given an input spike pattern x, and single target and actual output spike timest i and t i , respectively, the learning rule is applied as follows with learning rate η: This learning rule takes the difference between two spiketiming coincidence windows, driving a neuron to fire a single spike at its target timing via synaptic weight modification. Specifically, the first term on the right-hand side measures the ability of each input spike at t j to induce an output spike att i , while the second term measures the responsibility of each input spike at t j for causing an output spike at t i . Using this information, the rule changes weights in such a way as to suppress any inputs which contribute to an output spike occurring far fromt i , and to reinforce any inputs which will contribute to the emission of an output spike neart i . The rule also works to produce graded weight adjustments when t i is close tot i , resulting in smooth convergence towards a solution by gradually changing the moment at which the neuron's membrane potential crosses its firing threshold in response to input-evoked PSPs. In this work only the first output spike from each neuron is used for the learning window, although the FILT rule is capable of handling multiple spikes per neuron. In its original formulation the FILT rule was devised to train single-layer SNNs, and tested on randomly-generated input data.

B. Approximate Backpropagation using FILT
This work uses an augmented version of the FILT rule in order to tackle the MNIST classification problem using a multilayer SNN. In [14] the goal was restricted to training single-layer SNNs to learn associations between arbitrary input and output spike patterns. The goal in this work is to correctly classify input images of handwritten digits via a first-to-spike decision process in the output layer of a network, as facilitated using precise spike time learning in hidden and output layers. With the ultimate purpose of inducing the correct output neuron to spike earliest in response to its associated class of input data, we use the target spike timet l j of each neuron j in a layer l as a training parameter. Rather than being static as in [14], eacht l j is selected anew for each neuron in each training iteration according to two metrics. These metrics are: (1) the first actual output spike time t l j , if present, and (2) a figure of merit associated with each neuron which we term the "desirability" d l j . The desirability reflects a measure, relative to the other neurons in layer l, of how helpful an early spike from neuron j will be. The desirability of every neuron in each layer lies within the range [−1, 1] and is computed recursively with the output layer l = L as the base case. For classification, the output layer neuron which is assigned to the correct class has a default desirability of 1 while all other outputs have -1. For neurons in any non-output layer, 1 ≤ l < L, the desirability is calculated using the desirabilities of the neurons in layer l + 1 and the corresponding weights w l+1 ij . Specifically, the vector of pre-normalized desirabilities for all N l neurons in layer l isd l ∈ R N l , calculated as where w l+1 ∈ R N l+1 ×N l is the matrix of weights connecting layer l to layer l + 1, and d l+1 ∈ R N l+1 is the vector of desirabilities in layer l + 1. Bold symbols are used to indicate vector or matrix quantities. Thed l vector is then normalized: This shifts the desirabilities in layer l to fall within the range [−1, 1], preventing signal decay in a network with many hidden layers. The relative rankings of the neurons in layer l are preserved by this normalization. The lowestd l j in the layer becomes d l j = −1 and the highest becomes d l j = +1. The desirability of a given neuron thus gives a relative measure of how strongly that neuron will excite (suppress) neurons in layer l + 1 that themselves have a high (low) desirability.
The desirability rule is somewhat analogous to a typical SNN error backpropagation algorithm in that it contains a measure of how beneficial each neuron is given the desired outcome of the final layer. However, unlike true backpropagation the desirability rule does not explicitly compute the error gradient with respect to each weight. Instead a much simpler calculation is used, thereby saving significantly on computing power. This also equips the network to counteract the vanishing gradient problem for deep networks.
After computing d l , the neurons are all trained via the FILT rule (6) where d t is the desirability threshold. If t l j does not exist, it is substituted by 5·l ms. The purpose of the target shift term δt is to draw forward the spike times of the most desirable neurons to distinguish them from the others. In turn, the spike times of the neurons in layer l + 1 connected most strongly to the desirable neurons in layer l will be similarly drawn forward. Ultimately the ideal output neuron will be encouraged to spike earlier than its neighbors, thus correctly classifying the input. The impact of the target shift δt for simple systems will be examined in Sec. IV.

C. Additional Learning Constraints
To accompany the FILT rule we also apply RMSprop [18] for synapse-specific, adaptive learning rates, providing a means to modulate the magnitude of weight changes. RMSprop has successfully been applied to multilayer SNN training in [8], motivating our choice here. With a typical value of β = 0.9, we calculate the RMSprop scaling term R b for each batch b as where ∆w b is the set of all basic weight changes computed according to (6) in batch b and R 0 = 1. Then the weights w b are updated: where η = 0.01 and = 0.003 are the learning rate and RMSprop offset term, respectively. The offset is used to avoid singularities when R b has entries close to zero. Synaptic scaling is also applied to the hidden layers to ensure that a reasonable level of spiking activity is present. Without this the hidden units may stop spiking entirely and thus present no activity to excite subsequent layers. The final Fig. 1. (a) The predicted change in a neuron's membrane potential u i at its actual firing time t when trained with the FILT rule for a single iteration with δt = 0.5 ms. The neuron is stimulated by two input spikes occurring at 0 and 2 ms, each of which is received through a different synaptic weight. The terms ∆u 1 i and ∆u 2 i indicate the individual contributions to the net change ∆u i , which result from training on the first and second input spikes, respectively. An equilibrium point exists at t = t E when ∆u i is zero. (b) Two simulated spike rasters showing that the spike time of a single neuron trained on these inputs approaches t E whether the initial spike time is before or after t E , demonstrating that t E is an attractor point. Counterintuitively, t E is positively correlated with the δt. (b) t E for a two-input neuron with input spikes at [0, ∆t] as a function of δt where ∆t is a parameter. Provided that ∆t is not too much larger than δt so that there is some training window overlap, the resulting t E is positively correlated with ∆t. Overall, the greater the delay between inputs and the greater the training shift, the more delayed the equilibrium spike time will be. weights for each batch w b * are computed with an additional scaling term: where γ = 0.01 is the scaling coefficient, S b is a matrix containing the number of spikes produced by each neuron. The precise matrix manipulations required to match the dimensions of S b and w b are omitted for brevity. A dropout rate of 35 % is also applied to the network during training. Dropout has been shown to be useful in preventing overfitting [19].

A. Single Neuron Response
We first study the behavior of a simple one-neuron network as described in Sec. II B. The neuron is presented with one or more arbitrary input spike trains injected through two channels, and its weights updated according to the FILT rule as defined by (6) with a target shift term δt as in (9). Suppose each input channel contributes a single spike, with timings t 1 and t 2 for the first and second channels, respectively. The membrane potential of the neuron, excluding the reset kernel, is then described by If the neuron spikes at t i and its weights subsequently updated according to the FILT rule, with its target firing time shifted forwards by δt with respect to t i , then its membrane potential at t i changes by In Fig. 1(a) we show ∆u i (t i ) as a function of t i for a sample selection of spike times: t 1 = 0 ms and t 2 = 2 ms, and δt = 0.5 ms. This includes the individual contributions from the two input channels. There is a clear equilibrium point t E at which ∆u i (t i ) = 0, indicating that if t i = t E the membrane potential of the neuron will not change as a result of applying FILT. However, if t i > t E then ∆u i (t i ) will be positive, causing t i to decrease in the next iteration. Alternatively, if t i < t E then δu i (t i ) becomes negative, leading to an increase in t i . Thus we conclude that t E is an attractive equilibrium for t i under the FILT rule with positive δt. To test this conclusion we simulated several trials involving a single neuron with two input channels contributing spikes at [0, 2]ms and randomly initialized weights. Fig. 1(b) shows the spike rasters for the neurons in two such trials where the initial spike was either earlier or later than t E . In order to further understand the dynamics of a neuron being trained with a target shift, we calculated the predicted t E value for various δt, for both a single-input neuron and a two-input neuron with the second input spike occurring an arbitrary time ∆t after the first input spike (see Fig. 2). We note that, counterintuitively, the larger Fig. 4. Spike rasters for the input, hidden and output layers arising from two input images presented at 0 and 10 ms. A histogram of the spike timings for each layer is included as well.
the applied δt the greater t E becomes. Adding a second input spike has the effect of shifting the t E curve upwards when δt is sufficiently large that the corresponding single-input t E value occurs after the second input spike. Prior to this point, the second input has no influence. In Fig. 3 we show a neuron learning with two different input spike trains. Each input train has a unique t E to which the output spike time would converge if trained solely on that input. For each training iteration, one of the two spike trains is randomly selected and presented to the network as an input. Despite having to match two different input sets, the neuron does remarkably well in converging toward each t E in response to each input train, as indicated by the solid lines in the figure.
We believe this discussion serves to illustrate the potential usefulness of setting the target timet i = t i − δt term within the FILT rule. In many supervised spiking neuron training algorithms, a specific, absolute target time must be chosen. In any reasonable neuron model there will be some delay between a presynaptic spike and the corresponding PSP peak. With multiple inputs and multiple layers, it can therefore be difficult to judge the optimal time for each neuron to spike in response to its particular inputs even prior to considering proper network output. The target shift addition to the FILT rule allows each neuron to find its own optimal spike time in response to each different input pattern, similar to an unsupervised neuron. However we are also able, in a reinforcement manner, to  restrict the spiking of specific neurons by modifying their target time according to the desirability value as discussed above.

B. MNIST Classification
We now study the classification of images containing handwritten digits provided by MNIST. An example of the MNIST classification network output is shown in Fig. 4. The images of a '7' and a '6' are provided to the network at 0 and 10 ms respectively. The images are transformed into input spikes as described in Section II C., which mostly cluster near 0 and 10 ms. The hidden neurons encode this information more evenly across the timing windows. Despite learning rules which attempt to enforce one spike per output neuron, some images such as the ones tested in Fig. 4 result in multiple output spikes. Despite this the two samples provided to the network are both correctly classified. The 7th and 6th neurons, respectively, are the first to spike, as well as being the most active during their respective windows. We tested networks with multiple different hidden layer sizes, learning rates and d t thresholds. Each network was trained for 500 batches with batch sizes of either 20 or 100 samples each. There was no significant difference in results noted between the two batch sizes. For comparison, a standard fullyconnected non-spiking network of equivalent size was trained in TensorFlow using sigmoidal activation neurons which we treat as roughly equivalent to spiking neurons. This Tensorflow network utilized full error-backpropagation training. The mean test accuracy using ensembles of 10 networks with 100, 200 and 300 hidden units respectively is plotted against the batch number in Fig. 5 alongside dashed lines that indicate the final test accuracy of the TensorFlow networks. In these simulations d t = 0. The three trials very nearly approach the accuracy of the backpropagation networks. For 100 hidden units the optimal learning rate is shown to be 0.03 as indicated in Fig.  6(a) although the variation between η = 0.01 and η = 0.04 is slight. In additional trials that explored η values outside this range the performance dropped off sharply. In Fig. 6(b) we study the effect of varying d t for the 100-unit network. The best case appears to be a d t = −0.1, indicating that roughly the top 45th percentile of hidden neurons should be reinforced and the rest suppressed. Indeed the best trial matches or outperforms even the larger 300-unit backpropagation network shown in Fig. 5.

V. DISCUSSION
We have devised and demonstrated a new supervised learning rule for classifying images with spiking neural networks. We would like to begin the discussion by highlighting and reiterating several of its important aspects.
First, this rule begins with FILT, a single-layer supervised learning algorithm, and modulates it using the backpropagated desirability as a fitness signal. Imprecise backpropagation, relying on approximated feedback signalling, has previously been shown to be an effective learning method [20], [21]. The FILT rule, among some others like it, is derived using a rate-substitution method which solves the discontinuous spike-gradient problem. However unlike some other supervised spike-based learning rules such as [3], FILT preserves nonlinearity by assuming nonlinear spiking rate functions. In general, any suitable approximate backpropagated signal could be used in tandem with any functional single-layer learning rule for spiking neurons, leading to a new class of hybrid multilayer learning methods. This class possesses some of the advantages of both full SNN backpropagation methods and local learning methods.
Second, this hybrid class has significantly reduced computational complexity due to the lack of a true error function and accompanying gradient calculation. By using only implicit reference to network output, we produce a learning rule that would be especially helpful for deep networks and neuromorphic hardware.
Third, by beginning with a single-layer training rule that seeks to impose a specific output spike train on the neurons and usingt l j = t l j −δ t we allow the chosen neurons to find their own natural spike time. This maintains some of the degrees of freedom in the network while still imposing a desired pattern by selecting which neurons are trained according to a finitet l j via the d l j and d t values. This bears some relation to STDP, a timing-based rule that many SNN implementations in hardware employ [22]- [26]. Hardware built to natively implement STDP should therefore be more easily adapted to use the reinforced, timing-based FILT rule as opposed to a full backpropagation method. An example is Heidelberg's Digital Learning System (DLS) platform, which not only implements local, STDP-like, pre-and postsynaptic correlation traces, but also includes an embedded microprocessor dedicated to plasticity processing that is capable of combining local traces with a third factor, such as an external reward signal [25]. DLS also operates at a large speed-up factor of 1000 compared to the biological time-scale it emulates, making it ideally suited to long-term SNN simulations.
Interestingly, we observe that our hybrid learning approach shares certain similarities with a reward-maximization procedure as studied in [6], [27]- [29]. In more detail, we first note that the FILT rule approximates the supervised, maximumlikelihood approach to learning for a probabilistic neuron model, but taken in the limit of a deterministic system [14]. We then consider that, as described in [6], [29], if the target spike train of a neuron trained by maximum-likelihood is substituted with its actual output spike train, then the learning rule becomes an unsupervised one. Hence, if this unsupervised rule is combined with an external 'success signal' to guide weight changes, then the rule works in such a way as to maximize the likelihood of a neuron generating a spike train which positively correlates with the receipt of a positivevalued success signal: a process that is otherwise referred to as reward-maximization [29]. Taking these points into account, we relate our hybrid learning method based on FILT to the reward-maximization method for the following reasons: • In our implementation the target firing time of a neuron is not explicitly supervised: the prescribed target timing actually depends on the actual output firing time of the neuron, but shifted earlier by a small amount. • When unsupervised the rule works to progressively shift the actual firing time of a neuron earlier, but since the FILT rule is additionally modulated by an external 'desirability' signal that is linked to the overall success of the network, the firing time is instead adjusted to shift either forwards or backwards according to how this influences the network's goal. • Since the goal of the network is to drive an early response in one of the output layer neurons according to its associated class label, then the communicated desirability works to shift the strongest contributing hidden spikes earlier. • Although the evaluation of the modulatory signals used here involves more computational steps than those considered in [29], they essentially share the same end result of providing task-specific feedback in order to guide desirable weight changes.
Hence, it is for the above reasons that we can interpret our hybrid training method as driving a form of reward-maximization during network training, by a combination of layer-wise local learning factors modulated by external signalling. It is also notable that reward-modulation of local synaptic activity factors is considered a biologically-plausible hypothesis for learning [30], where for example the neuromodulator dopamine is hypothesized to encode a reward-prediction error signal in the brain [31].
In conclusion, we note that although our experiments demonstrated reasonably high accuracy of our hybrid training method on MNIST, this did not reach the levels of some other, more refined spiking classifier implementations such as in [7], [32]. However, we believe this sufficiently demonstrates proof of concept for the hybrid learning rule we pioneer herein. The results achieved here are a lower bound on the possible performance of the rule, as different choices of reinforcement signal and layer-wise learning rule and additional tuning of the parameters can further enhance the network accuracy.