{'fraction_non_alphanumeric': 0.03391191825442863, 'fraction_numerical': 0.0153049841595645, 'mean_word_length': 4.90566534914361, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modeling Associative Plasticity between Synapses to Enhance Learning of Spiking Neural Networks

Spiking Neural Networks (SNNs) are the third generation of artificial neural networks that enable energy-efficient implementation on neuromorphic hardware. However, the discrete transmission of spikes brings significant challenges to the robust and high-performance learning mechanism. Most existing works focus solely on learning between neurons but ignore the influence between synapses, resulting in a loss of robustness and accuracy. To address this problem, we propose a robust and effective learning mechanism by modeling the associative plasticity between synapses (APBS) observed from the physiological phenomenon of associative long-term potentiation (ALTP). With the proposed APBS method, synapses of the same neuron interact through a shared factor when concurrently stimulated by other neurons. In addition, we propose a spatiotemporal cropping and flipping (STCF) method to improve the generalization ability of our network. Extensive experiments demonstrate that our approaches achieve superior performance on static CIFAR-10 datasets and state-of-the-art performance on neuromorphic MNIST-DVS, CIFAR10-DVS datasets by a lightweight convolution network. To our best knowledge, this is the first time to explore a learning method between synapses and an extended approach for neuromorphic data.


Introduction
Artificial neural networks (ANNs) have achieved the state-ofthe-art performance in various tasks such as recognition and tracking [Lecun et al., 2015]. However, there are some serious concerns about their huge demand of computational resources and power consumption [Li et al., 2016]. Meanwhile, mammalian brains show it realistic to have superior cognitive abilities with extremely low power consumption [Akopyan et al., 2015].
Inspired by the learning mechanisms of mammalian brains, spiking neural networks (SNNs) have been considered as a promising model for artificial intelligence (AI) and theoretical neuroscience [Roy et al., 2019]. Unlike traditional neural The average response of CA1 pyramidal cells to a stimulus from point II is observed. The curve depicts the membrane voltage change after point I and point II are stimulated separately and simultaneously for ten minutes under control conditions. (referenced from [Barrionuevo and Brown, 1983].) networks, SNNs transmit discrete spikes rather than real values between neurons. Therefore SNNs consume orders of magnitude less energy by enabling event-based computation, which can be carried out on neuromorphic chips [Akopyan et al., 2015;Davies et al., 2018]. Furthermore, SNNs are computationally more powerful than ANNs as the third generation of neural network models theoretically [Maass, 1997].
However, the non-differentiability of spike makes it unsuitable for traditional backpropagation, and the complete brain learning mechanism is far from clear, which poses significant challenges to the high-performance learning mechanism [Roy et al., 2019].
Most successful learning methods of SNNs focus on learning between neurons, for example, brain-inspired biological methods [Zhang et al., 2018], backpropagation methods with surrogate gradient [Wu et al., 2021;Zheng et al., 2021], and other novel methods [Lagorce et al., 2017;Sironi et al., 2018;Cheng et al., 2020]. When these methods only concentrate on learning between neurons and ignore the influence between synapses, some important information is also overlooked, making it difficult for efficient learning.
In this paper, we propose an Associative Plasticity Between Synapses (APBS) learning method inspired by the physiological phenomenon of associative long-term potentiation (ALTP). As shown in Figure 1 (A), point I and point II are stimulated, respectively, and the response of cells to point II is recorded for ten minutes. The results in Figure 1 (B) illustrate that the response is only enhanced when points I and II are stimulated simultaneously, indicating that there is a learning relationship between synapses. We model the associative plasticity between synapses by extracting shared weights, which is biologically plausible and computationally efficient. The shared weights build connections between synapses of the same neuron and transmit information when synapses are stimulated concurrently, which complements the learning mechanisms of SNNs. Since the biochemical mechanisms of this phenomenon may be different, we refine several update curves of the shared weights. Furthermore, we propose a spatiotemporal cropping and flipping (STCF) method for neuromorphic data augmentation, which enhances the temporal generalization ability of SNNs. Finally, we conduct extensive experiments to demonstrate the robustness and effectiveness of our methods.
Our key contributions are summarized as follows: • We propose the APBS method inspired by the ALTP phenomenon, the first description of the interaction between synapses. By complementing the learning mechanism between synapses, APBS reduces information loss and improves accuracy.
• We propose the spatiotemporal cropping and flipping (STCF) method to enhance the temporal robustness of SNNs. To the best of our knowledge, it is the first data augmentation method suitable for neuromorphic data.
• We achieve superior performance on static CIFAR-10 datasets and the state-of-the-art performance on neuromorphic MNIST-DVS, CIFAR10-DVS datasets with a lightweight convolution network.
2 Related Work

Learning algorithm of SNNs
With the vigorous development of SNNs, various algorithms have been proposed to explore an efficient and robust SNNs learning algorithm, mainly divided into two categories. The first is converted SNN, which converts some pretrained non-spiking neural networks into SNNs with the same architecture. For example, [Ding et al., 2021] proposes a rate norm layer to replace the ReLU activation function to get greater accuracy and better associate ANNs with SNNs.
[Han et al., 2020] proposes a "soft reset" spiking neuron model, referred to as residual membrane potential (RMP) spiking neuron, which retains the "residual" membrane potential above the threshold at the firing instants. The details of the APBS learning mechanism, which extracts shared weights from original weights.
The second is to train SNNs directly. For example, [Lagorce et al., 2017] and [Sironi et al., 2018] use (averaged) time surfaces structures to process the temporal information of SNNs. Based on spike-timing-dependent plasticity (STDP) rule [Bi and Poo, 1998], [Zhang et al., 2018] uses brain-inspired principles to train SNNs. Furthermore, the method of finding an approximate gradient for nondifferentiable spikes has become very popular in recent years. Among these methods, [Zheng et al., 2021] performs backpropagation (STBP) along with the spatial and temporal domains of SNNS. In contrast, [Wu et al., 2021] uses the accumulated spikes as an alternative gradient to avoid the huge memory consumption brought by the STBP method. Our method falls into this category.

Neuromorphic Data augmentation
Data augmentation is an explicit form of regularization that is also widely used in the training of ANNs. The two most popular and effective data augmentation methods in ANNs are random cropping [Krizhevsky et al., 2012] and random flipping [Simonyan and Zisserman, 2015]. Random flipping randomly flips the input image horizontally, while random cropping extracts a random sub-patch from the input image. To the best of our knowledge, augmentation methods for neuromorphic data remain a blank slate to be explored.

Method
This section first briefly introduces the spiking neuron model and then analyzes the modeling approach of APBS. Methods for neuromorphic data augmentation are subsequently introduced. Finally, the training process of the network is shown.

Spiking Neuron Model
The dynamics of a biological neuron are shown in Figure 2 (A). The neuronic membrane potential increases with the accumulation of weighted spikes, and an output spike is generated once the membrane potential exceeds a threshold. The widely used leaky integrate and fire model describes the dynamics of the membrane potential as: where τ m = R m C m is the membrane time constant, R m and C m are resistance constant and capacitance constant, respectively. u is the membrane potential, I(t) is the input current, V th and V rest are the spiking threshold and resting potential. Once u reaches V th at t i , a spike is generated and u is reset to V rest , which is usually taken as 0. The output spike o is described by the Dirac delta function δ(x). The input will be summed to I(t) by the dendrite response weight w. To facilitate the calculation and simulation, Equation (1) is converted into a discrete form by Euler method. Given that the time interval dt is constant during simulation, 1 − dt/τ m and 1/C m are absorbed into response weights w and leakage coefficient λ respectively. The discrete form of Equation (1) can be described as: where n denotes the n-th layer and w ij is the synaptic weight from the j-th neuron in pre-layer n to the i-th neuron in the post-layer n + 1. x i denotes the weighted input and H(x) is the Heaviside step function.

Associative Plasticity between Synapses
Motivation. The associative long-term potentiation phenomenon in Figure 1 illustrates that repeated stimulation of one synaptic afferent of a neuron can enhance the synaptic potential induced by another afferent stimulus of the same cell [DAVISON, 1985]. In short, there is an associative enhancement effect between synapses when stimulated concurrently. On the other hand, the associated long-term depressive phenomenon [Stanton and Sejnowski, 1989] corresponding to the ALTP phenomenon is also observed in hippocampal slices. Both of these phenomena indicate that there is associative plasticity between synapses. Modeling. We model the associative plasticity between synapses inspired by the shape and connection of neurons. As shown in Figure 2 (B), the original weight is divided into two parts in series, one is the independent neuronic coefficient w corresponding to the dendritic terminal, and another is the extracted shared weightŵ corresponding to the dendritic trunk. In this way, the w emphasizes the effect of learning between neurons andŵ emphasizes mutual influence between synapses. The original weight w n+1 ij is formulated by: where f is a function describing the change curve of the shared weightŵ. dw n+1 ij and dw n+1 ij are the gradients of w n+1 ij and w n+1 ij respectively. Above formulae fit well with the phenomenon in Figure 1 for three reasons: • Equation (3) indicates the neuronic coefficient w and shared weightŵ jointly determine the input weights. While in Figure 1 (B), influences from other neurons and other synapses corporately determine the response to input stimuli.
• Equation (4) suggests that gradients do not change when propagating back along the shared weight. Corresponding to Figure 1 (A), influences between synapses can not change the backward information.
• Equation (5) denotes that the shared weight is determined by the sum of neuronic weights. As shown in Figure 1 (A), interactions between synapses are induced by stimulation from other neurons.
Furthermore, since the biochemical mechanism of APBS is still unclear, we refine several update curves of the shared weightsŵ to explore the efficient form of function f . The simplest is f (ŵ) =ŵ, a monotonically increasing linear function. Besides, considering that the shared weight should be positive as a strengthening or debilitating factor, the absolute value function f (ŵ) = abs(ŵ) is adopted. Last but not least, the well-known ReLU function f (ŵ) = relu(ŵ) is introduced because the non-monotonic absolute value function may make learning turbulent. Experiments in Section 4 show the high performance of these curves.

Spatiotemporal Cropping and Flipping
Neuromorphic data are captured by event-based cameras, which only record dynamic events in the field of view, leading to severe overfitting while reducing the amount of information. Generally, each sample is represented by a number of events, which are described as: where t i is the timestamp at which the event is generated, x i and y i are the position of the pixel grid. p i is the polarity of the event, a binary value meaning respectively OFF and ON events, and I is the number of events. Inspired by the idea of crop and flip [Krizhevsky et al., 2012;Simonyan and Zisserman, 2015], we propose a spatiotemporal cropping and flipping (STCF) method for event data, which can be formulated as: where d i represents a generic expression for x i , y i , and t i while [D A , D B ] is the corresponding cropping range randomly generated. c ∈ {0, 1} indicates the random flipping parameter, set as 1 when flipping. The cropping area is first moved to the origin through d i − D A , and then flipped according to the parameter c. As shown on the left of Figure 3, the STCP method enhance the generalization of the network in the time domain.

Training Details
The forward process. The architectures we utilized are similar to other spiking convolutional neural networks [Lee et al., 2020;Wu et al., 2021;Wu et al., 2019]. As illustrated in Figure 3, event data is cropped and flipped by Equation (7), and then downsampled to input spikes, which are weighted chronologically accumulated to the membrane potential of spiking neurons. The dynamics of the neuron are described by Equation (2). The output spikes are also fed into the next layer in chronological order, and the cycle repeats. The expanded section in the middle of Figure 3 illustrates the direction in which data travels through space and time. Since the gradients are only passed at the end, it does not keep the computation graph every step like spacetemporal backpropagation. We measure the mean square error between the results and label vector Y within a given time window T, which follows the setting in [Lee et al., 2020;Wu et al., 2021]. The loss function is defined as: where the V th is the threshold and u N,T denotes the membrane potential vector of the output layer in Equation (2). The backward process: For the backpropagation method, the main target is to measure the gradient of each synaptic weight without omission accurately. We adopt the method of accumulating spiking flow [Wu et al., 2021] as the baseline and supplement the mutual influence between synapses with the proposed APBS method. Therefore we use the accumulated output-input spikes ratio as a gradient. The ratio S n i can be formulated by [Wu et al., 2021]: (11) It is worth mentioning that when the network is deployed on the chip, we can directly combine f (ŵ i ) and w ij into w ij . Thus the APBS learning mechanism will not add any operation in inference. In addition, signals in the network are transmitted in the form of spikes, including the input and pooling layers. Thus all the multiplications of the network can be replaced by logical AND &, resulting in SNNs consuming orders of magnitude less energy than ANNs [Roy et al., 2019]. Besides, it should be noted that in our model, biases are not introduced for calculation.

Experiments
In this section, we conduct extensive experiments to demonstrate the superior performance of the proposed methods. For  Table 1: Performance comparison of the proposed method and the state-of-the-art methods on CIFAR10-DVS, MNIST-DVS and CIFAR10 datasets. The * symbol means that this is the baseline we reproduced. FE is short for feature extraction layer and Resg is the graph resnet block. And the best results are in bold for clarity.
a fair comparison, we use the same hyperparameters and experimental settings with [Wu et al., 2021], which is our baseline. Specifically, the weights of the convolutional and fully connected layers are initialized by a normal distribution. The threshold and leakage coefficients of neurons are belonged [0.9, 2]. Adam optimizer is introduced to adjust the learning rate which is initially set to 5 × 10 −4 . Moreover, we apply the APBS learning mechanism on just one layer of neurons, as shown in Figure 3.

Datasets
We evaluate the performance of our methods on both neuromorphic datasets (MNIST-DVS, CIFAR10-DVS) and static datasets (CIFAR10). Barranco, 2013]. It contains three scales (4, 8, 16) of the digits, each having 10,000 samples. The recordings contain noise, blur, and other factors caused by a fixed DVS camera when capturing the moving original digit images. We use scale 16 of the datasets and perform training with 90% of randomly chosen samples while testing with the remaining 1,000 samples. Time steps and batch size are 30 and 40, respectively.

MNIST-DVS datasets [Serrano-Gotarredona and Linares-
CIFAR10-DVS datasets [Li et al., 2017]. It consists of 10,000 samples from CIFAR10, taken by DVS camera like MNIST-DVS. It is a challenging recognition task due to the complexity of the samples. We follow previous works [Wu et al., 2021;Zheng et al., 2021] by randomly selecting 90% of the images as the training images and the rest images are testing images for every class. Furthermore, we downsample input images to the resolution of 42 × 42 as in previous work [Wu et al., 2021]. Time steps and batch size are 100 and 40, respectively.
CIFAR10 datasets. It is very famous and widely used in deep learning for object classification. It contains 60,000 color images in 10 classes, of which 50,000 images are for training, and 10,000 images are for testing. We utilize a standard data augmentation strategy consisting of the random crop, random horizontal flip, random erase, and normalization. Poisson sampling is also applied for input. Time steps and batch size are 100 and 50, respectively.

Comparison with the State-of-the-Art
As shown in Table 1, we achieve the state-of-the-art performance on neuromorphic datasets and superior performance on static datasets. On the MNIST-DVS datasets, our model gets the classification accuracy of 99.6% or 0.4% error rate. It should be pointed out that although [Sironi et al., 2018] only uses a four-layer structure, it needs to calculate averaged time surface, which consumes more computing power. On the CIFAR10-DVS datasets, it can be seen that our model achieves 76.1% accuracy, which outperforms other methods by a large margin. It is necessary to point out that the baseline only achieves 62.5% accuracy, and our methods deliver a 13.6% improvement. The bionic APBS method is also applicable to static datasets. On the CIFAR10 datasets, we achieve an accuracy of 91.81%, which is also higher than other works with similar architecture sizes. Experimental results demonstrate that our methods are effective regardless of neuromorphic or static datasets.

Ablation Studies
We perform ablation experiments on different datasets to evaluate the effectiveness of each proposed component.

Overall Ablation Studies
On neuromorphic datasets, the previous work [Wu et al., 2021] is replicated as our baseline, then the APBS learning method and STCF method are respectively added to it. As shown in Table 2, the APBS method is effective on different bases, and the STCF method achieves a noticeable effect since it is the first neuromorphic data augmentation method.
It is worth noting that on the CIFAR10-DVS datasets, the combined effect of our methods is better than the sum of the individual effects.
Ablation Studies on STCF Captured by fixed event-based cameras, neuromorphic data records the changing points in the field of view, allowing for better capture of critical information. While the amount of information plummeted, it is more accessible to overfitting. As shown in Figure 4 (A), our model with VGG-7 architecture quickly achieves a full score on the training set, but is unable to fit more test set data earlier. We naturally propose the STCF method, the first neuromorphic data augmentation method. Figure 4 (B) illustrates the effect of STCF method.
As expected, the model learns more slowly on the training set while degrading less on the test set. Since the STCF method increases the diversity of the spatiotemporal data, the generalization performance of the network is significantly improved. Besides, we also note that the STCF method is orthogonal to most existing methods, thus providing a higher baseline.
Ablation Studies on APBS As described in Section 3.2, different variation curves of shared weights are evaluated on CIFAR10-DVS datasets with the Resnet-9 network. The control variable method is used for a fair comparison, and all configurations are the same except the APBS method. The curves, including f (ŵ) =ŵ,  f (ŵ) = abs(ŵ) and f (ŵ) = relu(ŵ), are marked as APBSx, APBS-abs, APBS-relu, respectively. As shown in Figure 5, the blue lines represent APBS methods and the red lines show the control group without APBS methods. It turns out that the performances of these curves are comparable and significant, indicating that the APBS method is robust and effective. Furthermore, multiple architectures are introduced to evaluate the robustness of the methods, including the most popular residual network structure. Table 3 shows that the APBS learning methods and STCF method are also effective and robust for different network structures.

Conclusion
In this work, we propose the associative plasticity between synapses (APBS) mechanism for SNNs. It is inspired by the associative long-term potential phenomenon observed in the mammalian brain. By making complementary modifications between synapses, APBS captures more learning signals and strengthens the ability of SNNs to fully explore spatiotemporal information. In addition, we propose a spatiotemporal cropping and flipping (STCF) method for neuromorphic data augmentation. The proposed methods achieve the state-ofthe-art performance on the neuromorphic datasets, which indirectly indicates the biological plausible of methods. Since the methods are orthogonal to existing methods, they provide a better baseline for future research. We are excited about the future of the APBS mechanism and STCF data augmentation method and plan to generalize them for other vision tasks. We believe that associative plasticity between synapses can enhance learning in SNNs, which will surely become a pillar for further exploration of brain-like computing.