{'fraction_non_alphanumeric': 0.03915674563936242, 'fraction_numerical': 0.010876873788711782, 'mean_word_length': 4.6869095816464235, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks

In this era of machine learning models, their functionality is being threatened by adversarial attacks. In the face of this struggle for making artificial neural networks robust, finding a model, resilient to these attacks, is very important. In this work, we present, for the first time, a comprehensive analysis of the behavior of more bio-plausible networks, namely Spiking Neural Network (SNN) under state-of-the-art adversarial tests. We perform a comparative study of the accuracy degradation between conventional VGG-9 Artificial Neural Network (ANN) and equivalent spiking network with CIFAR-10 dataset in both whitebox and blackbox setting for different types of single-step and multi-step FGSM (Fast Gradient Sign Method) attacks. We demonstrate that SNNs tend to show more resiliency compared to ANN under black-box attack scenario. Additionally, we find that SNN robustness is largely dependent on the corresponding training mechanism. We observe that SNNs trained by spike-based backpropagation are more adversarially robust than the ones obtained by ANN-to-SNN conversion rules in several whitebox and blackbox scenarios. Finally, we also propose a simple, yet, effective framework for crafting adversarial attacks from SNNs. Our results suggest that attacks crafted from SNNs following our proposed method are much stronger than those crafted from ANNs.


I. INTRODUCTION
Contemporary machine learning models like Artificial Neural Networks (ANN) have completed several milestones towards gaining super-human performance in visual recognition tasks like image classification, text and voice recognition [1], [2] etc. Application of such networks is being considered for autonomous cars, drones and robotics. For such mission critical applications, there is an urgent need to improve the robustness of networks against adversarial attacks. Adversarial attacks [3], [4] can be generated by injecting carefully-crafted perturbations to a clean input so that it can deceive the model into producing incorrect outputs with high probability. Note, the perturbation is small enough to be perceptible to the human eye. This vulnerability holds even when the adversarial input is generated from a different trained model other than the target This work was supported in part by C-BRIC, one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA. [5]. The profound implication of the problem has triggered research interest towards addressing this issue and finding ways to defend against adversarial attacks in the context of state-of-the-art neural network models.
Research in [3], [4], [6], [7] shows the vulnerability of deep ANNs against adversaries. Several adversarial training and defense mechanisms have been proposed in this regard, like ensemble training [8], implicit prior modeling with random noise [9], scalable training [10] etc. However, one fundamental question that remains unanswered is whether there are any network models inherently resistant to adversarial attacks. In the face of this question, more biologically plausible neural network model, like Spiking Neural Network (SNN) comes into picture. In an SNN, the network receives stochastic stimulation from noisy neurons in the form of Poisson spike train, leading to the temporal evolution of the membrane potential of the neurons [11]. This inherent noise embedded in an SNN makes it worthwhile to investigate how the spiking network reacts under adversarial attacks, compared to ANNs.
In this paper, we have analyzed the behavior of large-scale spiking neural networks against state-of-the-art adversarial attacks. Our experiment is focused on VGG-9 network models with CIFAR-10 dataset. To the best of our knowledge, this is the first work to analyse the characteristics of a spiking network under different kinds of adversarial attacks. The key contributions can be summarized as follows: • Spiking Neural Network model is a comparatively newer addition to the machine learning family. Although there exists significant amount of literature on the technique of crafting adversarial input from ANN, there is none to generate SNN-crafted adversary. We propose a simple mechanism to generate adversarial inputs from SNN model parameters without the need of any non-trivial gradient calculation in the spiking domain. • We present comprehensive quantitative comparison of the behavior of ANN and equivalent SNN under different attack scenarios. We observe that spiking networks are more robust than rate-based ANNs for blackbox attacks (i.e. when attacker has no knowledge of the target model's parameters). In whitebox attack (i.e. when attacker has full knowledge of the target model's parameters), SNNs, generally, yield higher accuracy degradation than ANNs. Furthermore, our results suggest that attacks crafted from SNNs following our proposed method are much stronger than those crafted from ANNs. • We demonstrate that the adversarial resistance of SNNs varies depending on the training mechanism. We have considered two different training methods: ANN-to-SNN conversion [12], [13] and direct spike-based backpropagation [14], [15] in our experiments. We observe that the latter method shows better resistivity under whitebox and blackbox scenarios.
The organization of the rest of the paper starts with illustration of the basic concepts of the adversarial attacks and Spiking Neural Network (SNN) in section II and III, respectively. In the next section (section IV), we explain the network architecture, training methods, adversarial input generation and testing process. Section V contains our simulation and analysis results, followed by conclusion in section VI.

II. ADVERSARIAL ATTACK: FUNDAMENTALS
Given a classification model h with dataset (x, y true ), where x is the clean image and y true is the corresponding correct label, the main concept of adversarial attack is to find an input x adv such that x and x adv are indistinguishable to the human eye, yet the model misclassifies x adv , i.e. produces high probability on wrong labels. In our work, we have considered the following approaches to generate x adv .

A. Non-targeted FGSM (Fast Gradient Sign Method)
This is the most basic and widely used approach to generate adversarial perturbations in the following form [4] x adv = x + sign(∇ x J(x, y true )) Here refers to the amount of perturbation. Usually, the value of is much smaller than the unperturbed data x. J(x, y true ) is the loss function of the model. ∇ x J is the gradient of the loss function with respect to the original clean data.

B. Non-targeted R-FGSM (Random-step FGSM)
In this method, the single step gradient calculation is preceded by a simple step of adding small random noise (N (O d , I d )) to the image beforehand.
Here, initial perturbation α < . Authors in [8] introduced this method to escape the non-smooth vicinity of the data point.

C. I-FGSM (Iterative FGSM)
This is a multistep method for generating adversarial inputs. It iteratively applies FGSM with step-size α ≥ /k, where k denotes the number of iterations [8]. In non-targeted I-FGSM, the loss is calculated with respect to the true label y true , whereas targeted I-FGSM uses either a random class, y random or the least likely class, y ll for calculating loss function and perturbs the input in the opposite direction as the gradient.
1) Non-targeted I-FGSM: x N +1 adv = Clip x, {x N adv + αsign(∇ x J(x N adv , y true ))} (5) 2) Targeted I-FGSM: x N adv is the adversarial sample at N th iteration. y true and y ll are the correct and the least-likely class label, respectively. α is the perturbation per step. Clip x, () denotes element-wise clipping of the argument to the range [x− , x+ ]

III. SPIKING NEURAL NETWORK: FUNDAMENTALS
Spiking Neural Networks (SNN) operate based on bioplausible event-driven algorithm. From network topology perspective, the activation blocks (like Rectified Linear Unit) of the ANN is replaced by biological neuron-based functional blocks (e.g. Integrate and Fire (IF) neuron, Leaky Integrate and Fire (LIF) neuron) in the equivalent SNN. The dynamics of LIF spiking neuron is formulated as follows: V mem is the membrane potential of the neurons, τ is the time constant for the decay of V mem , w is the synaptic weight and θ(t − t k ) represents the spike at time instant t k . There are mainly two broad categories for training an SNN: unsupervised and supervised. However, in this work, we have used two of the supervised training strategies in order to achieve high accuracy. A brief illustration of these two techniques are presented in the next two subsections.

A. ANN to SNN conversion (SNN-I)
ANN to SNN conversion method considers a simple Integrate and Fire neuron (IF) as the neuron activation function due to its functional resemblance to Rectified Linear Unit (ReLU), without any leak or refractory period. Several authors [12], [13] have proposed techniques of adjusting the synaptic weights ('weight normalization') or neuronal threshold values ('threshold balancing') to acquire lossless transformation from ANN to SNN. The accuracy reported for SNNs, trained in this way, is high, compared to ANN, even for very large scale networks.

B. Spike-based training (SNN-II)
In this method, SNN is directly trained based on an eventdriven supervised gradient descent backpropagation algorithm. Unlike the conversion mechanism, LIF neurons are used as the basic building block here. In forward propagation, Poissondistributed spike train, generated from the input pixels, are fed to the network. Accumulated weighted spikes at the input of a neuron, at time t, triggers an output spike, if it exceeds a threshold value. Neurons at each layer undergo this process based on the input spikes received from the preceding layer. In order to carry out backpropagation in the spiking domain, we need a differentiable transfer function for the neurons. To that effect, the activation of the spiking neuron is formulated by low-pass filtering the spike train, according to the following equation.
The time constant τ dictates the decay rate of the neuron activation. T is the total time. t k refers to the time instant of the k-th spike. During the backpropagation process, the gradient of error with respect to weight requires calculating the derivative of the neuron activation a with respect to the net input to the neurons, which is approximated by the following equation: where V th refers to the threshold value of the neuron, net is the accumulated weighted sum of spikes at the input of a neuron, and t is the time instant. The details of the backpropagation algorithm is illustrated in [14], [15].

A. Dataset and network topology
Our experiments mainly focus on the standard visual recognition dataset CIFAR-10 with VGG-9 networks. The VGG-9 architecture is 32×32-64c5-64c5-2s-128c5-128c5-2s-256c5-256c5-256c5-2s-1024fc-10o, where c = convolutional layer, s = sub-sampling layer, fc = fully connected layer and o = output layer. The input image in CIFAR-10 dataset results in 3-channel 32×32 input neurons, followed by 2 subsequent layers of 64 convolutional kernels of size 5×5 each, followed by 2×2 spatial averaging sub-sampling window. This convolution process is repeated in the second and the third stage with 128 and 256 maps of convolutional kernels, respectively. Note that the third stage has 3 convolutional layers. The final two stages of the networks are fully connected layers. The outputs from the third stage sub-sampling is vectorized and fed into a fully connected layer with 1024 outputs. The final layer consists of 10 output neurons corresponding to the 10 classes of CIFAR-10. It is worth-mentioning that each of the convolutional, subsampling and fully connected layers are followed by LIF neurons (ReLU activations) in SNN (ANN) architecture. We have also used a dropout of 0.2 after each convolution and fully-connected layer.

B. ANN Training
The first step of our experiment consists of training the ANN models (network topology described in the previous section), as showed in Fig. 1. Training of VGG-9 ANN is performed with 200 epochs at an initial learning rate of 0.09, which is reduced by a factor of 10 at the 81 st and the 122 nd epoch (also known as learning rate annealing) in order to ensure gradual decrease of the loss function during the training process. Our custom simulation framework is implemented based on PyTorch deep learning library [16].

C. SNN Training
In order to train the spiking VGG-9 model by following the conversion method [12] (SNN-I), we have adjusted layer-bylayer neuronal threshold values (theory in sec III-A) with the maximum membrane potential at the corresponding input, by running the forward propagation sequentially for each layer. We have used a total of 2000 time steps for the entire forward propagation, since it demands a sizable time-window to find the optimum threshold values.
On the other hand, in case of the spike-based backpropagation training of SNN (SNN-II), total number of time steps is 70 during the forward propagation. The training is performed with 125 epochs, where the learning rate is reduced at the 70 th and the 100 th epoch. The details of the training theory can be found in sec. III-B and Ref. [15]. The baseline accuracy of the ANN and the trained SNNs are summarized in table I.

D. Adversarial input generation: ANN-crafted
After the completion of the training phase, adversarial inputs are generated from the trained models using four different methods: (i) Non-targeted FGSM, (ii) Non-targeted R-FGSM, (iii) Non-targeted I-FGSM and (iv) Targeted I-FGSM. The flowchart in Fig. 1 elaborately describes the FGSM method of adversary generation. According to (1), we have calculated the sign matrix from the input gradient and multiplied with   Fig. 1. A schematic summarizing the training, adversarial input generation and testing methods used in our work for both Artificial and Spiking Neural Networks (ANN and SNN). Training: ANNs are trained according to the standard backpropagation formula, while two different techniques have been followed for training SNNs: ANN-to-SNN conversion [12], [13] and direct-spike based backpropagation [14], [15]. The generated SNNs are referred as SNN-I and SNN-II, respectively. Adversarial input generation: The flowchart inside the dotted red box, labelled as ANNAdv, refers to the Fast Gradient Sign Method (FGSM) [4]. According to this method, adversarial input for M AN N is produced from the clean input perturbed by the sign of the gradient of the loss function with respect to the input (∇ X J). is the amount of perturbation. In case of generating adversary from SNNs, first, we have created a transformed ANN (M T AN N ) by loading the weights (or scaled weights, in case of SNN-I) of the trained SNNs into a randomly initialized ANN. Next, rate-encoded input Xrate is passed through the ANNAdv method, with M T AN N . Dotted green box is used to represent SNN-crafted adversary generation method, labelled as SNNAdv. perturbation . Since we normalized the image dataset to represent zero mean, the absolute values of the input pixel intensity in clean images range from 0 to 1. Hence, the amount of perturbation inflicted upon the pixels needs to be normalized too. We have used = 8/255, 16/255, 32/255 and 64/255 (in some cases) in our experiment. In the Iterative FGSM, we have experimented with two-step, five-step and seven-step iterative methods in order to investigate the effect of the number of iteration steps. Note that we have used a random class as y ll in (7) for Targeted I-FGSM. All of the parameters used in (1) - (7) have been summarized in table II.

E. Adversarial input generation: SNN-crafted
For a comprehensive analysis and comparison, we have devised a method to generate attack samples from SNNs as well. Algorithm 1 describes the widely-known FGSM in the context of ANN. Algorithm 2 illustrates its proposed SNN counterpart. FGSM calculates the gradient of the loss function with respect to the clean input data. Due to the non-trivial operations during gradient calculation in a spike-based model, we have come up with a simple framework. Initially, an ANN model, N N with the same network topology is randomly initialized. The SNN model is independently trained (M SN N ). Subsequently, N N weight matrices are mapped and overwritten with the learned weights of M SN N . Next, rate-based input, X rate is generated from the Poisson spike train of the clean dataset. Afterwards, X rate and N N model are used to generate FGSM adversarial input following Algorithm 1. This method has been Adversarial data:X i adv = AN N Adv(X i rate , N N , ) 9 end schematically illustrated in Fig. 1. Note, for SNN-I, the scaling factors of the weights of the transformed ANN, M T AN N (Fig.  1) equal the threshold-scaling factors used in the ANN-to-SNN conversion mechanism during training.

F. Testing
The last stage of our experiment consists of testing ANN, SNN-I and SNN-II with different types of adversarial inputs. The adversarial inputs are passed through the forward function of the networks and compared against the true labels to compute the corresponding adversarial test accuracy and loss. We have performed four different sets of comparisons, as described in the "Testing" section in Fig. 1

A. ANN versus SNN adversarial robustness
First, we compared the ANN and SNN behavior under whitebox scenario (column 1 of Fig. 2), where each target network is attacked by adversarial inputs created using the target's parameters (ANN is attacked by ANN-crafted, SNN-I by SNN-I crafted and so on). It is evident from column 1 in Fig. 2 that ANN faces smaller degradation in accuracy compared to SNN-I and SNN-II against different kinds of whitebox attacks (Non-targeted FGSM, R-FGSM, I-FGSM scenarios) for varying ranges. For instance, in FGSM whitebox attack ( Fig. 2(a) column 1), when = 8/255, O 1 (ANN loss), O 2 (SNN-I loss) and O 3 (SNN-II loss) correspond to 27.8%, 43.5% and 35.5%, respectively. However, we observed that with a targeted attack (specifically, targeted I-FGSM shown in Fig. 2 (d) Column 1), SNN-I and SNN-II losses are lower than ANN.
Next, we compared the robustness of the models against blackbox attacks. It is worth mentioning that ANN and SNN models differ in terms of network dynamics and adversarial input generation mechanism. Hence, to have a fair comparison in blackbox scenario, we used a common source model (separately trained and different from the target model) to generate  In each of these cases, the amount of perturbation has been varied from 8/255 to 32/255. Column 1 presents whitebox attack results. Columns 2-4 present blackbox attack results, wherein the networks are attacked by a common set of adversarial inputs (SNN-I crafted blackbox adversary for column 2, SNN-II crafted blackbox adversary for column 3 and ANN-crafted blackbox adversary for column 4). Note, the method detailed earlier in Fig. 1 is used to generate SNN-crafted adversary from SNN-I and SNN-II. The loss for each network is calculated as the difference between the baseline accuracy (clean input test) and test accuracy (adversarial input test). Smaller loss in accuracy implies more adversarial robustness. Minimum loss values are highlighted with red rectangles in each subplot.

ANN-crafted Blackbox
the adversaries and subsequently, attacked ANN, SNN-I and SNN-II with it, as illustrated in Fig. 2. The source model used to generate adversarial example is SNN-I, SNN-II and ANN in column 2, 3 and 4, respectively. Interestingly, in contrary to the whitebox results, we observe that SNNs turn out to be more robust in blackbox setting. Even for the ANN-crafted blackbox attack case (column 4), where all three networks yield smaller accuracy degradation, SNNs show lower loss than ANN. This points to the fact that spike-based computing with temporal dynamics has some intrinsic resistance, compared to standard rate-based ANN dynamics. We conjecture that the stochasticity in the temporal dynamics that is inherited with spike computing might be contributing to this adversarial resistance. Another noteworthy observation here is that the accuracy loss observed with SNN-I/-II-crafted blackbox attacks across all the models is significantly higher than ANN-crafted blackbox attacks. This implies that adversarial inputs generated with temporal spike-based SNNs cast stronger attacks than ratebased ANNs. Revisiting the whitebox attack results, we can surmise that the stronger attack created from SNN models is the cause of higher accuracy degradation for spiking models in that case.
In summary, we can deduce the following from the above results with regard to ANN vs. SNN adversarial effects: 1) SNNs cast stronger adversarial attacks than ANNs, leading to more accuracy loss for SNN, compared to ANN, in whitebox scenario. 2) SNNs are more robust in blackbox setting than ANNs due to the inherent stochastic temporal dynamics.

B. Dependence on SNN training method: SNN-I versus SNN-II
Now, we analyse the dependence of the training mechanism (used to create the SNN) on its adversarial resistance. In order to obtain a clearer picture, we compared the adversarial resistance of SNN-I (ANN-to-SNN conversion) and SNN-II (direct spike based training) against different kinds of attacks, described as follows (Fig. 3 , we observed that the accuracy loss of SNN-I was significantly higher than SNN-II. Hence, in this case, one might expect that SNN-II target models will yield very high accuracy loss compared to SNN-I targets (since the attack is crafted from SNN-II). However, the amount of accuracy degradation for SNN-II is still comparable to that of SNN-I. This suggests that networks trained with spike-based training have more adversarial resistance that conversion-based models. We speculate that the deterministic nature of conversion based models (converting ReLU values to IF functionality) does not entirely inherit the stochasticity in temporal dynamics causing adversarial susceptibility for SNN-I models. • ANN-crafted Blackbox (BB3): In this scenario, the common adversarial source model is an ANN. Here the loss values for SNN-I and SNN-II lies within a range of 1% for all attack types and are much lower than the BB1, BB2 scenarios. This further corroborates the fact that SNNs craft stronger attacks than ANNs. Note, Fig. 3 basically contains the adversarial test results for SNN-I and SNN-II presented in the previous figure (Fig. 2) at a fixed value ( = 16/255).

C. Dependence on I-FGSM iteration steps for SNN-II
In order to investigate the effect of the number of iteration steps used in I-FGSM on SNN, we have plotted the loss of SNN-II for Non-targeted and Targeted I-FGSM attacks with two-step (black), five-step (blue) and seven-step (red) iterative methods in Fig. 4(a) and (b). Note, iterative attacks are stronger than single-step attacks (such as, FGSM or R-FGSM). I-FGSM input for the target model (M SN N −II ) has been created from three different source networks: M SN N −II (whitebox: solid lines), M SN N −II x (SNN-II-crafted-blackbox: dashed lines) and M AN N x (ANN-crafted blackbox: dotted lines). We note that the step-number variation has little effect on the performance of the models in all cases. In addition, as observed earlier, SNN-crafted blackbox inputs cause stronger attacks compared to ANN-crafted blackbox. To analyse this further, we compared the slope of the lines in Targeted I-FGSM attack ( Fig. 4(b)). Amount of loss rises from 6% to only 22% for ANN-crafted blackbox, as we increase (the amount of adversary) from 8/255 to 64/255, whereas loss due to SNN-crafted blackbox attack (dashed line) undergoes a staggering increase from 6% to 50%. This establishes the effectiveness of SNN-based adversaries for casting stronger attacks.

VI. CONCLUSION
In this paper, we analyzed the role of bio-plausible Spiking Neural Networks in the domain of adversarial attacks. As an initial work in this field, we have addressed some unexplored issues like introducing simplified method for crafting adversarial sample from spiking networks and dependence of robustness on SNN training mechanism. In addition, our quantitative comparison for variety of adversarial attacks presents a comprehensive picture of the performance of these networks under different attack scenarios. Finally, the key findings and recommendations from our analysis are: • SNNs craft stronger attack than ANNs in both whitebox and blackbox setting. • While SNNs undergo higher accuracy degradation than ANNs in whitebox scenarios, they yield significantly higher resistance than ANNs for blackbox attacks. The temporal dynamics and inherent stochasticity in SNNs might be responsible for such behavior. Further work is required to understand the role of temporal dynamics for adversarial resistance.
• SNNs trained with spike based backpropagation are more robust than SNNs obtained from conversion rules against both whitebox and blackbox attacks. This further ascertains the role of stochasticity (preserved in the spikebased backpropagation mechanism) to strengthen adversarial resistance.