{'fraction_non_alphanumeric': 0.03340981236445925, 'fraction_numerical': 0.0137033661250275, 'mean_word_length': 4.81469298245614, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sparse Adversarial Video Attacks with Spatial Transformations

In recent years, a significant amount of research efforts concentrated on adversarial attacks on images, while adversarial video attacks have seldom been explored. We propose an adversarial attack strategy on videos, called DeepSAVA. Our model includes both additive perturbation and spatial transformation by a unified optimisation framework, where the structural similarity index (SSIM) measure is adopted to measure the adversarial distance. We design an effective and novel optimisation scheme which alternatively utilizes Bayesian optimisation to identify the most influential frame in a video and Stochastic gradient descent (SGD) based optimisation to produce both additive and spatial-transformed perturbations. Doing so enables DeepSAVA to perform a very sparse attack on videos for maintaining human imperceptibility while still achieving state-of-the-art performance in terms of both attack success rate and adversarial transferability. Our intensive experiments on various types of deep neural networks and video datasets confirm the superiority of DeepSAVA.


Introduction
In the past decade, Deep Neural Networks (DNNs) have demonstrated their outstanding performance in various domains, such as image classification [36,53], text analysis [26], speech recognition [8], and object detection [8,61]. Despite their huge success in these tasks, recently some researchers have shown that DNNs are surprisingly vulnerable to adversarial attacks [2,32,41,59], e.g., adding a small human-imperceptible perturbation to an input image can fool DNNs, enabling the model to make an arbitrarily wrong prediction with high confidence [33,41]. This is raising serious concerns about the readiness of deep learning models, especially on safety-critical applications such as face authentication [27], surveillance systems [31], and medical applications [36]. Hence, it is of vital importance to investigate the performance of DNNs on adversarial examples and evaluate their robustness in an adversary environment [34,38,56].
As such, significant research efforts have emerged to assess the robustness of DNNs under adversarial attacks, notable works include Fast Gradient Sign Methods (FGSM) [59], C&W attack [2], DeepFool [28], and JMSA [50]. These attack strategies primarily concentrate on image-related tasks, yet the adversarial robustness of deep learning models on videos has not been comprehensively explored. Recently, a number of works [15,29,47,57] are aware of the values of the adversarial attacks on videos. Theoretically, attacking videos is more challenging compared with images because videos contain temporal information. So video attack not only requires to achieve minimal adversarial distance but also needs to perturb as few frames as possible. As such, identifying the most effective frame(s) and generating  competitive perturbation upon those frame(s) are of huge importance to the success rate of the attack. Another important consideration is the efficiency. As perturbing each frame of the video is time-consuming, we expect to perform the influential-frame identification and adversarial perturbation simultaneously so we can maintain human imperceptibility and achieve high attacking success rate. In practice, DNNs processing videos are widely applied in real systems such as video surveillance [31], action recognition [17] and autonomous driving [52]. In particular, most of those applications directly relate to the decisions concerning property security or human health and safety. As a result, investigating adversarial samples on videos is urgently needed. However, to achieve a high-performance adversarial attack on videos, the following challenges need a careful treatment: Maintaining the fidelity of the produced adversarial examples with real-world scenarios: One important criterion for adversarial attacks is that the perturbed example should resemble a real-world instance as close as possible. Current video attack strategies all adopt the l p -norm metric to measure the fidelity of the perturbed examples. Although l p -norm is effective to capture noise contamination, it is sensitive to naturally-occurred transformations such as rotation, spatial shift, and scaling [63]. Taking Figure 1 as an example, a slight rotation or scaling of pixels will lead to an obvious difference in l p -norm distance. Thus, attacks constrained by l p -norm cannot capture some spatial transformations that naturally happen in a real-world scenario such as the shaking, vibration, or rotation of a camera. Achieving a high attacking success rate without compromising the human imperceptibility: Different to static images, videos contain sequential data structure and change dynamically with the temporal dimension. Hence, the existing attack strategies designed for images are not directly applicable to videos. Perturbing all frames of a video, although could achieve a high fooling rate, is time-consuming and also potentially compromises human imperceptibility. Thus, perturbing as few frames as possible while maintaining a high attacking success rate is highly desired on adversarial video attacks, which can be tackled by sparse attacks.
Enabling the adversarial video attack to be effective across diverse types of DNNs: In a real-world scenario, we may not be able to access the parameters, structures, or even datasets of a pre-trained DNN. Thus, similar to adversarial attacks on images, a strong adversarial transferability that can work across diverse unseen models is desirable. However, unlike DNNs for images that are without temporal structure, video models are more complicated and include diverse neural units for recurrent operations. Hence, achieving a satisfying adversarial transferability across unseen models is also a non-trivial challenge.
As a result, we provide a pioneering exploration to deal with the above challenges. We propose an adversarial video attack for DNNs, called DeepSAVA, which can i) capture a wide range of adversarial instances including both noise contamination and various spatial transformations; ii) achieve sparse attack, i.e., only perturbing very few frames of a video while still achieving a state-of-the-art attack success rate; and iii) obtain better adversarial transferability across various recurrent models compared with baseline methods.
In summary, there are three key contributions in DeepSAVA: Structural Similarity Index Measure (SSIM): instead of l p -norm, we adopt the SSIM metric in the loss function to constrain the distance between adversarial and clean videos. As demonstrated by the community of Image Quality Assessment, SSIM is an alternative signal fidelity measure that is superior to l p -norm on some applications where human perceptual criterion matters [63]. As Figure 1 shows, SSIM is less sensitive to both noise and spatial transformations such as rotation and scaling, which is more resemble human perception.
Combination of additive perturbation and spatial transformation: we are the first work to combine additive and spatial-transformed perturbation for video attacks. According to the image attack used spatial transformation perturbation [55,60], changing the positions of pixels could improve perceptual realism and make it locally smooth. In DeepSAVA, we introduce a new term in the loss function for optimising both additive and spatial transformation perturbation. With a proper SSIM-based constraint, we could produce strong perturbations combined with additive and spatial transformation. Such combined perturbation enables DeepSAVA to achieve successful attacks by just perturbing one frame.
Novel alternating optimisation strategy: we are also the first work that uses Bayesian optimisation (BO) to choose the most critical frames of the video in attacks. To achieve a video attack that can perturb as few frames as possible, we design an alternating optimisation strategy that can effectively identify the key frames via BO and then initiate additive and spatial-transformed perturbations on the selected key frames by stochastic gradient descent (SGD) based optimiser. Such an alternating process happens in each iteration of the optimisation until key frames are found. Combining the above two ingredients, the proposed novel optimisation strategy could achieve a better fooling rate than baselines.
The flow chart of our method is illustrated in Figure 2. We anonymously release the code of DeepSAVA 1 and generated adversarial videos across multiple models 2 .

Related Work
Video Action Recognition Models: The video classification task primarily focused on action recognition [20]. The works on video classification using DNNs are developed in two ways: using 2D or 3D-based convolution neural networks (CNN). Since the CNNs have obtained state-of-the-art performance on image classification, Karpathy et al. [18] first proposed to use 2D CNN to classify each frame of the video. Szegedy et al.then developed the Inception-v3 [39,40], which is commonly used as a baseline classification model. As 2D-CNNs use incomplete video information, some works added layers containing temporal information such as LSTM to integer CNN features extracted over time which is referred to as CNN+LSTM model [6,30]. As for the 3D CNNs [43], it can learn temporal features from videos by inputting all frames in three dimensions directly. Vadis et al. [3] proposed a two-stream inflated 3D CNN (I3D) to build the 2D kernel first and then merge the pooling layer and kernel into a 3D network. By pre-training the I3D on Kinetics Dataset, it could reach state-of-the-art performance on recognising UCF101 and HMDB51 action video datasets.
Adversarial attack on images: The adversarial attack on images has been explored extensively recently. Szegedy et al. [41] first proposed to add visually imperceptible noise on the images to mislead pre-trained CNN to give the wrong prediction label. Goodfellow et al. [10] proposed to use a gradient-based approach, the fast gradient sign method (FGSM), to generate adversarial examples. DeepFool [28] is then proposed to find the minimal perturbation by iteratively linearizing the loss function. Other gradient-based optimisation algorithms to generate perturbation were also proposed [2,25,42,54]. These works mentioned above only apply additive perturbation on pixels. Some works [16,22,23,51,55] use a functional perturbation which is non-additive-only perturbation like spatial transformation. These perturbation slightly perturb the location of pixels. Some works such as [11,16,62] also utilize other types of metrics such as SSIM to quantify the human perception, but none of them explored the SSIM-guided spatial transformation. For more details on adversarial attacks, please refer to our recent survey [13] and tutorial [1,34].
Adversarial attack on videos: Wei et al. [47] claimed that they are the first to attack videos. Instead of attacking each frame of a video, they apply additive perturbations on randomly selected frames and use l 2,1 norm to guide the Flickering [29] RL [57] Heuristic [48] Append [4] BlackBox attack [15] GAN-based attack [24] Sparse Attack [ Table 1: Comparison with related work in different aspects.
(a) (b) Figure 3: (a) The process to perturb one frame of a clean video X, where the first frame is masked to be perturbed by spatial flow vector U and noise N. (b) The systematic optimisation process by using Bayesian Optimisation and Adam Optimiser.
gradient-based optimisation and evaluated the performance on the CNN+LSTM model. Li et al. [24] used a GAN network to generate offline universal perturbations for each frame. Chen et al. [4] proposed to append a noise frame to the end of videos, which is obtained based on all videos. Naeh et al. [29] applied flickering temporal perturbations on each frame to generate universal perturbations for the I3D model. Jiang et al. [15] were the first to propose a black-box approach to attack videos. Wei et al. [48] proposed to use a heuristic method and Yan et al. [57] used a reinforcement learning algorithm to select the key frames to perform black-box attack. However, these works only applied additive perturbation based on l p -norm distance. In Table 1, we compare our method with existing related works on video attacks in six aspects. Our work applies the SSIM-guided non-additive perturbation on selected frames to generate adversarial videos efficiently. We also propose a novel alternating optimisation strategy to select the key frames.

Methodology
Problem Definition: The video classifier is defined as J(·; θ) with pretrained weights θ. The input clean video is defined as X = (x1, x2, ..., xT ) ∈ R T ×W ×H×C , where T is the length of the video (number of frames), and W, H, C represents the width, height, and the number of channels of each frame; its adversarial video generated is represented asX. In order to obtain the adversarial example, the original video is perturbed by a spatial transformer S, and additive noise D. Given that the ground truth label of input video X is y, the objective function is: where 1y is the one-hot encoding of y; similar is the similarity loss function to measure the distance between generated adversarial and original video; adv is the loss function to measure the difference between ground truth and prediction label. The parameter λ is set to balance these two loss terms. Additionally, the cross-entropy is used to calculate the adv , which is proved to be effective in [47].

Sparse Spatial Transform Adversarial Attack
Structural Similarity Index Measure (SSIM): The SSIM was first proposed in [44], and is detailed in [45]. It measures the local similarities between the local pixels on three aspects: structures, contrasts, and brightness. As we mentioned before, the SSIM is less sensitive to the combined perturbation and more similar to human perception than lp-norms [63]. As the SSIM is differentiable with respect to the input variable (the definition and derivation process of SSIM are shown in Appendix A), we apply SSIM to calculate the similarity loss to constrain the perturbation during the optimisation process. The overall SSIM score for the video is calculated by summing up the SSIM loss over all frames.
Sparse Attack: The mask indicator M = (m1, m2, .., mT ) ∈ R T is used to choose the key frames in the video, where mt ∈ {0, 1} indicates whether the t-th frame is masked to be perturbed. The masked video Xm is formed through the map function M(M, X), and then fed into the spatial transformer S.
Spatial Transformed Perturbation: Given the t-th frame x t ∈ R W ×H×C of input video X, x t n denotes the n-th pixel of x t and its location in the frame can be represented by a 2D coordinate (h t n , v t n ). The spatial transformer [14] S is a differentiable model composed by flow displacement vectors U = ((∆H 1 , , which is used to synthesize the 2D coordinate of adversarial videos. Supposex t n with location (ĥ t n ,v t n ) is the adversarial example transformed from x t n , given its corresponding spatial displacement vector (∆h t n , ∆v t n ), the new location of original pixel x t n can be represented as (h t n , v t n ) = (ĥ t n + ∆h t n ,v t n + ∆v t n ). Considering the sparse attack mask indicator M , we can represent the transformed adversarial video asXS = S(U, X, M ).
Additive Perturbation: The additive perturbation is the most common way to generate adversarial examples [2,10]. We define the additive model as D with parameter N ∈ R T ×W ×H×C . We combine spatial transformation and additive perturbation to generate adversarial videos as (illustrated in Figure 3

Novel Alternating Optimisation Strategy
In this paper, we utilize the Bayesian Optimisation (BO) to select the most critical frames. As the frame selection is a discrete variable optimisation problem, we also tried other discrete optimisation techniques such as simulated annealing (SA) [9] and genetic algorithms (GA) [49], but both spent about 200s to find the final result which is much longer than about 16s taken by BO.
The generated adversarial video is formed asX = N · M + S(U, X, M ). In this paper, the similarity loss similar and adversarial loss adv in problem (1)  As M is a discrete binary vector, which makes problem (4) non-differentiable, we solve it systematically by a novel alternating optimisation strategy: at each iteration, we optimise M by BO first; and then by fixing M , the problem becomes differentiable, which can be solved by Stochastic Gradient Descent (SGD) based optimisation. We choose the Adam optimiser [19] because of its robust and fast convergence performance. This process repeats for a fixed number of iterations, continuously improving the solution via both techniques alternatively. During the BO process, we will find the best mask indicator through several iterations. In the k-th iteration of BO, we first sample a candidate M k according to the acquisition function F . Then, the corresponding loss L k will be computed by the Adam, which will then affect the next sampled point M k+1 for the next iteration. When the BO reaches the maximum exploration number, the best M with minimum loss will be fed into the Adam optimiser to generate the final adversarial video. The process is illustrated in Figure 3

Experiments
Dataset: As action recognition video datasets are widely used in adversarial video attack studies, we choose two popular benchmark action recognition datasets to evaluate the performance of our method: UCF101 [37] and HMDB51 [21]. Both datasets are realistic action recognition datasets. The UCF101 contains 13,320 videos with 101 categories such as playing instruments, body movements, human-object interaction. Similarly, the HMDB51 has around 7,000 videos within 51 categories related to body-motion and facial actions.
Action Recognition Models: We evaluate DeepSAVA on three classifiers: Inception-v3, a 2D-CNN based model [40], which is widely used in the image recognition task with high accuracy; I3D, a 3D-CNN based model, pre-trained on Kinetics [3]; CNN with LSTM, which is pre-trained on ImageNet to extract features from videos and then input these features to train the LSTM network.
The training accuracy of all classifiers is shown in Table 2.
Baseline methods: Two baseline methods are used for comparison, the Sparse [47] and Sparse Flickering. For the works shown in Table 1, only [47] is the white-box sparse attack; [48][57] are black-box sparse attack methods. As our work is a white-box sparse attack, we choose the most related one, Sparse [47], as the main baseline. We perform perturbation directly on the frame, while [4] appended additional frame in the end of video, which is more visible to human. So we did not include it as a baseline due to its compromise on the similarity of human perception. In [24], GANs are used to attack real-time video, which is not comparable to our method. We modified Flickering [29], which perturbs all frames, into a sparse one as the Sparse Flickering baseline, but we still show the performance of perturbing all frames.

Comparison With Baseline Methods
In this section, we will show the comparison results between DeepSAVA and baselines. Since running BO will add extra time to choose the frame, to make the comparison more complete, we also take the DeepSAVA without BO selection into account.
Limited iterations: Since each method uses a different metric, in order to control the maximum allowed perturbation we limit the number of search iterations for all methods. Each iteration only allows a small amount of perturbation (controlled by the learning rate of Adam optimiser), following the same setup used by the baselines. The results in Table 3 show that the ANIs are much below    We run the experiments 10 times, and show the average results with a 99% confidence interval. For the methods without frame selection, the first frame is perturbed.As shown in Table 3, BO selection is more efficient than the one without BO. This happens because it is able to select the most critical frame, which can improve the efficiency on most of the cases. For the CNN+LSTM model, DeepSAVA increases the FR slightly compared with the baselines; while for the I3D model, we can see that the FR grows significantly. The BO selection process is also essential for I3D. Without BO, only about half of the test videos can be attacked successfully; after applying BO, the FR increases to nearly 100%. As for the Inception-v3 model, the FR increases when applying DeepSAVA. It can be concluded that the CNN+LSTM is the most robust classification model among the three classifiers. Although the I3D has the highest classification accuracy, it is more vulnerable to attacks, even when only one frame is modified. That might happen because the I3D model relies heavily on the integral structure of the video itself and some frames may be more important.
We find that the position of key frames is related to the classifiers evaluated: for CNN+LSTM, the frames in the front are more often selected, and for other CNN networks, the position is variant. Thus, it is reasonable that the BO cannot improve the FR for CNN+LSTM model as much as the I3D, as we attacked the first frame when not selecting. We also show the results in Figure 4 for attacking a different number of frames across I3D and CNN+LSTM models. It can be seen that the more frames attacked, the higher the fooling rate obtained.
Fixed l2,1 norm and SSIM: For the purpose of a fair comparison, we also present the results under fixed l2,1 and SSIM budgets for perturbing only one frame. The maximum allowed iteration is set to be 500 to limit the time. As the baseline methods are based on lp-norm and our method is on SSIM, we take experiments under the same lp-norm constraint and SSIM constraint, respectively. Based on the results of fixed iterations, we randomly select 200 videos from different categories to attack the I3D model on the UCF101 dataset. During the experiments, the Sparse Flickering spent days to achieve the constraint, thus we will only compare with the Sparse [47] attack. In [7], the SSIM budget for attacking image is set to 0.95, thus we choose the SSIM constraints above 0.95. In [58], it states that the difference between the images is imperceptible when the l2,1 score is 4, given that, we also set the l2,1-norm budget to below 0.1 (since 0.1 * 40 = 4, as we have 40 frames). As we can see in Table 13, under small fixed budgets, DeepSAVA outperforms the Sparse [47] in both cases in terms of FR and total time.   Table 6: Comparison with Sparse baseline, DeepSAVA without BO and with BO on different models by only perturbing one frame. Gray cell shows the best results.

Average Absolute Perturbation
Average Absolute Perturbation (AAP) is introduced to measure the perturbation level for each method. As mentioned previously, the sparse Flickering adds a small perturbation per frame, but cannot obtain comparable results to ours. Thus, we choose the pure sparse attack (Sparse) as the main baseline to show the average absolute perturbation. As the baseline is guided by l1,2 norm and ours is based on SSIM loss, we will record the average perturbation of l1,2 and SSIM separately. To achieve a fair comparison, we set the maximum l1,2 norm ball constraint as 0.1 and maximum SSIM constraint as 0.92. Suppose the fooling rate is f , and distant matrix is D, which can be set to (1-SSIM) or l1,2 norm, thus the average absolute perturbation(AAP) can be represented as: where V adv denotes the generated adversarial video that could successfully mislead the classifier and Dmax is the maximum constraint; N is the number of adversarial samples achieving successful attack. We run experiments on 200 random selected videos of UCF101 dataset and record the results of FR, ANI, AAP(l1,2) and AAP(SSIM) in Table 6. From the results, we can see that for the model I3D and Inception-v3, our method could achieve better performance in terms of efficiency, fooling rate and AAP(SSIM). For the CNN+LSTM model, our method engages higher fooling rate and spends less time, and the AAP(l1,2) and AAP(SSIM) are also acceptable compared with the baseline model.

Visualization of Results
The generated adversarial frames by DeepSAVA are presented in Figure 5. Because of the spatial transformation, the frame looks a little bit shaky but not obvious in human eyes. In fact, in the real world, it is normal to see that there are a few frames with instabilities during video shooting and transmitting. That's why we apply the spatial transformation in video attacks to improve the efficiency and fooling rate. In practice, a distortion in one frame of a video is less noticeable than a static image since this specific frame only appears for 0.047 seconds in human eyes [54]. We could also see that it does not lead to a noticeable perturbation as shown by our video demos.
When transmitting the videos in the real world, the generated frames need to be compressed into videos first and then decompressed to frames. We found that for the additive-only perturbed frames, they may not remain adversarial examples after such transmission. Our experiments demonstrate that DeepSAVA can be immune to short video compression due to the fact that perturbation based on spatial transformation can be well preserved during compression while additive perturbation may disappear.  Table 7: Effects of combining noise (D) and spatial transformation (S) by modifying a different number of frames on UCF101; Mask N means that N frames are modified.

Ablation Study
We perform ablation experiments to study the effects of combined perturbation for a different number of attacked frames by comparing with additive noise only and spatial transform only perturbations, and the effects of BO selection. Table 7 shows the FR for three classifiers on the UCF101 dataset. Four approaches are taken to attack the model: 1) only noise (D), 2) only spatial transformation (S), 3) combination of additive perturbation and spatial transformation (D + S), and 4) combined perturbation with BO selection. To make more comprehensive evaluations on the superiority of combination, we attack a different number of frames for the CNN+LSTM model as it has the lowest FR when only perturbing one frame. All experiments showed the combination power to increase the FR; using BO selection is also useful, especially for the I3D model.

The Accuracy of Bayesian Optimisation Selection
To justify whether the Bayesian Optimisation could select the most critical frames, we take the brute force search experiments to obtain the upper bound of the performance: when the selection frame is 1, we select the key frame manually one by one of the video, and then record the maximum loss found by the search. We randomly select 100 videos from UCF101 in different categories. The fooling rates, average maximum loss and average time spent for one video on three models are shown in the    [35], LSTM [12], and GRU [5] networks respectively. The training accuracy for vanilla RNN and GRU are 65.16% and 73.05% respectively.
As Figure 4 shows that the Sparse [47] performs better than the Sparse Flickering in terms of FR, we choose the Sparse [47] as the baseline method. The fooling rates (FR) of the generated videos across models are presented in Table 9. The models in rows are used to generate adversarial videos and the models in columns are the target attack classifiers. Here we disturb seven frames of a video to enlarge the attacking success rate. We use the adversarial examples generated from the white-box attack for the transferability, which leads to the FR in the diagonal being 100%. These adversarial examples are then used to attack other models (like a black-box attack) as detailed in Table 9. Comparing with the baseline, our approach has a higher FR which indicates a better performance in terms of transferability. The difference between vanilla RNN and the other models is that vanilla RNN has no memory component, so it shows a weak performance on the video classification task. As we observed, adversarial videos generated from LSTM and GRU models can fool the vanilla RNN successfully. Additionally, the FR across GRU and LSTM are around 85%, which shows good transferability between the recurrent models with memory. However, from the fooling rates shown in Table 9, we could see that the transferability from RNNs to CNNs is not as good as that from CNNs to RNNs. the fooling rates to attack RNN models are higher than those to attack the I3D and Inception-v3 models. While that happens maybe because the I3D model has the highest training accuracy, it engages the lowest fooling rate when performing the black-box attack on it. As we can also conclude that due to the lowest training accuracy the Vanilla RNN model obtains, it achieves the highest fooling rate on attacking unseen Vanilla RNN model. Overall, compared with the Sparse baseline, our method could achieve better transferability.

Conclusion
In this paper, we apply spatial transformed perturbation and additive noise to attack as few frames as possible to obtain the sparse adversarial videos. We run experiments on the UCF101 and HMDB51 action dataset and three models. The most influential frames to be attacked are selected by a joint optimisation strategy with Bayesian optimisation (BO) and SGD-based optimisation. Additionally, the quality of generated adversarial examples is measured by SSIM instead of lp-norm. We obtain better results than state-of-the-art sparse baselines in terms of both fooling rate and transferability. Our most significant results are for the I3D model, by only attacking one frame of the video to obtain 99.5% to 100% attack success rate.  have N pixels in the total videos, the SSIM can be represented as: where xi andxi are the i-th pixel of each frame in the video. To apply the gradient decent optimisation method described in Section 3, we have to compute the gradient of SSIM with respect to the adversarial video exampleX. As equation (9) shows, to compute ∇X SSIM(X,X), we only need to calculate the gradient ∇x i SSIM(xi,xi). The process is represented as follows. [46] We define four parameters to deduce the derivative of local SSIM: Therefore, the gradient can be expressed as:

B Effects of λ
To decide the value of λ, we applied the DeepSAVA without BO selection on 200 random selected videos of UCF101 dataset to evaluate the effect of λ. The average success perturbation (ASP) is the average of the SSIM score of perturbation for the adversarial examples that could mislead the model successfully: ASP (SSIM )) = avg(SSIM (V adv − V original )), where V adv denotes the generated adversarial video that could successfully mislead the classifier and V original is the original video. The results of applying λ = 0.8, 1.0, 1.5 on three models are presented in Table 10. We can see that the bigger the λ, the lower the FR while the lower the perturbation.While, for the CNN+LSTM model, the fooling rate remains the same across all tested λ values, but the perturbation level is the lowest at λ = 1.5. Thus, we choose λ = 1.5 for the CNN+LSTM model and λ = 1.0 for I3D and Inception-v3 model to trade off the performance in terms of the fooling rate and average success perturbation.
C Comparison Experiments with l p -norm and SSIM Constraints C.1 I3D model    For the CNN+LSTM model, we can see that although the Sparse baseline could obtain higher fooling rate, but it will spent much more time to generate the adversarial examples. Our method using combined perturbation will spent less time and obtain comparable fooling rate. Because here we select the first frame to attack, the BO cannot improve the performance as I3D model.