{'fraction_non_alphanumeric': 0.037230694747034616, 'fraction_numerical': 0.01457274267731784, 'mean_word_length': 4.754422621535033, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Boosting Fast Adversarial Training with Learnable Adversarial Initialization

Adversarial training (AT) has been demonstrated to be effective in improving model robustness by leveraging adversarial examples for training. However, most AT methods are in face of expensive time and computational cost for calculating gradients at multiple steps in generating adversarial examples. To boost training efficiency, fast gradient sign method (FGSM) is adopted in fast AT methods by calculating gradient only once. Unfortunately, the robustness is far from satisfactory. One reason may arise from the initialization fashion. Existing fast AT generally uses a random sample-agnostic initialization, which facilitates the efficiency yet hinders a further robustness improvement. Up to now, the initialization in fast AT is still not extensively explored. In this paper, we boost fast AT with a sample-dependent adversarial initialization, i.e., an output from a generative network conditioned on a benign image and its gradient information from the target network. As the generative network and the target network are optimized jointly in the training phase, the former can adaptively generate an effective initialization with respect to the latter, which motivates gradually improved robustness. Experimental evaluations on four benchmark databases demonstrate the superiority of our proposed method over state-of-the-art fast AT methods, as well as comparable robustness to advanced multi-step AT methods. The code is released at https://github.com//jiaxiaojunQAQ//FGSM-SDI.

the examples added with imperceptible perturbations can easily fool well-trained DNNs. It has been proven that DNNs' realworld applications [6]- [10] are also vulnerable to adversarial examples. Adversarial examples thus pose a huge threat to the commercial applications of deep learning. Improving the model robustness against adversarial examples is a challenging and important issue. A series of defense methods [11]- [21] have been proposed since then, among which adversarial training (AT) [12] has been proved to be among the most effective ones by injecting adversarial examples during training. In detail, the AT methods adopt the model gradient information to generate adversarial perturbation and then add the generated adversarial perturbation to the original clean sample to generate adversarial examples. AT methods can be formulated as a minimax problem [12], [22] with the inner maximization maximizing the loss to generate adversarial examples and the outer minimization minimizing the loss on the generated adversarial examples to obtain a robust model. The robustness depends on the inner maximization [23], i.e., the adversarial example generation. But the generation of the adversarial examples is an NP-hard problem [24], [25]. Thus AT methods always adopt the model's gradient information to generate adversarial examples [5], [12].
Based on the number of steps in generating adversarial examples, AT can be roughly categorized into two groups, i.e., multi-step AT [12], [26]- [28], and fast AT [29]- [33]. Multistep AT adopts multi-step adversarial attack methods such as projected gradient descent (PGD) [12] and achieves comprehensive robustness in defending against various attack methods. However, they require a high computational cost to perform multiple forward and backward propagation calculations in generating adversarial examples. To boost training efficiency, fast AT methods are proposed, which need to calculate gradient only once and adopt fast gradient sign method (FGSM) [5]. Although they can greatly reduce time and computational cost, the robustness is far from satisfactory, compared with other state-of-the-art multi-step AT methods. Therefore, plenty of studies have explored how to improve the robustness of fast AT. Among them, some studies [31], [32] focus on the initialization issue, as it is proved that using a random initialization in fast AT plays an important role in improving robustness [32]. However, the diverse random initialization fashions adopted in existing fast AT methods are usually sample-agnostic, which restricts further robustness improvement.
To overcome such a shortcoming, in this paper, focusing on image classification, we propose a sample-dependent adversarial initialization to boost FGSM-based fast AT, dubbed FGSM-SDI. The sample-dependent initialization is calculated by a generative network conditioned on not only a benign image  [12], FGSM-RS [30], and our FGSM-SDI in the loss landscape of binary classification. Background is the contour of cross entropy. The redder the color, the lower the loss. PGD-AT is a multi-step AT method that computes gradients w.r.t the input at each step. FGSM-RS uses a random sample-agnostic initialization followed by FGSM, requiring the computation of gradient only once. But our FGSM-SDI uses a sample-dependent learnable initialization followed by FGSM.
which refers to the original clean image without adversarial perturbations, but also its signed gradient from the target network. The benign image provides position information in the loss landscape, while the signed gradient provides a rough direction of increasing the loss. The initialization is then exploited by FGSM to generate a final adversarial example for training. The pipeline of adversarial example generation of our FGSM-SDI is illustrated in Fig. 2. Note that the generative network and the target network are jointly learned under a minimax optimization framework, where the generative network attempts to create an effective initialization for FGSM to fool the target network while the target network improves its robustness against such adversarial examples via adversarial training. Along with an increasingly robust target network, the generative network dynamically optimizes a more effective initialization for FGSM, boosting the robustness of fast AT. Fig. 1 presents the differences between a typical AT method (PGD-AT [12]), a fast AT method (FGSM-RS [30]), and our FGSM-SDI in generating adversarial examples. PGD-AT is an advanced multi-step AT method that can achieve decent performance in robustness but is time-consuming in calculating gradient at multiple steps. FGSM-RS calculates gradient only once at x b0 to which the benign image x is moved with a random sample-agnostic initial perturbation. Differently, our FGSM-SDI calculates gradient at x c0 to which the benign image x is moved with an optimized sampledependent initial perturbation. Our learnable initialization depends on the benign image as well as its signed gradient, which is more informative than the random one. Specifically, we adopt a lightweight generative network to generate the different adversarial initialization for different sample inputs, i.e., samplespecific initialization. And the generative network uses not only sample information but also sample gradient information to generate the adversarial initialization. In this way, compared with the random initialization, the proposed initialization is more informative. For PGD-AT, our FGSM-SDI can achieve comparable robustness with a much more efficient training process. Compared to current fast AT methods (e.g., FGSM-RS), our FGSM-SDI outperforms them in robustness by a large margin, though with a slight sacrifice on efficiency due to the additional generative network. Note that such an additional generative network is in fact lightweight and acceptable. (see results in Sec. IV-E) Our main contributions are summarized as follows: • We propose a sample-dependent adversarial initialization method for fast AT. The sample-dependent property is achieved by a generative network trained with both benign examples and their gradient information from the target network, which outperforms other sample-agnostic fast AT methods. Our proposed adversarial initialization is dynamic and optimized by the generative network along with the adjusted robustness of the target network in the training phase, which further enhances adversarial robustness. • Extensive experiment results demonstrate that our proposed method not only shows a satisfactory training efficiency but also greatly boosts the robustness of fast AT methods. That is, it can achieve superiority over state-ofthe-art fast AT methods, as well as comparable robustness to advanced multi-step AT methods.

II. RELATED WORK
In this section, we first introduce the related researches on attack methods. Then we introduce the related researches on defense methods, especially the adversarial training variants. Specifically, in this paper, we focus on the image classification task, where adversarial examples can fool a well-trained image classification model into outputting the erroneous prediction with a high level of confidence. Given a clean image x with the corresponding true label y and a well-trained image classifier f (·), the attack methods are used to generate the adversarial example x adv , to deceive the classifier into outputting an erroneous prediction, i.e., f (x adv ) = f (x) = y, where the distance function satisfies L p (x adv , x) ≤ , where represents the maximum perturbation strength and L p represents the distance between the adversarial image x adv and the clean image x under the L p distance metric, where p ∈ {1, 2, ∞}. In the recent researches of attack methods, L ∞ is a commonly used distance metric, which is also adopted in our paper.

A. Attack Methods
Szegedy et al. [5] [41] propose two parameter-free attack methods, i.e., auto PGD with cross-entropy (APGD-CE) and auto PGD with the difference of logits ratio (APGD-DLR), to overcome the problem caused by the suboptimal step size and the objective function. Moreover, they combine the proposed attack methods with two existing attack methods, i.e., FAB [42] and Square Attack [43] to form the ensemble AutoAttack (AA). Furthermore, AA has achieved state-of-the-art performance in evaluating the model robustness against adversarial examples.

B. Adversarial Training Methods
Adversarial training (AT) variants have been widely accepted to improve adversarial robustness under comprehensive evaluations. They can be formulated as a minimax optimization problem, i.e., the inner maximization maximizes the classification loss to generate adversarial examples and the outer minimization minimizes the loss of generated adversarial examples to train parameters of a robust model. Given a target network f (·, w) with parameters w, a data distribution D including the benign sample x and its corresponding label y, a loss function L(f (x, w), y), and a threat bound , the objective function of AT can be defined as: where the threat bound can be defined as = {δ : δ ≤ } with the maximal perturbation intensity . The core of the adversarial training is how to find a better adversarial perturbation δ. Typical adversarial training methods usually adopt a multi-step adversarial attack to generate an adversarial perturbation δ, i.e., multiple steps of projected gradient ascent (PGD) [12]. It can be defined as: where Π [− , ] d represents the projection to [− , ] d and δ t+1 represents the adversarial perturbation after t + 1 iteration steps. In general, more iterations can boost robustness in adversarial training due to generating stronger adversarial examples. The prime PGD-based adversarial training framework is proposed in [12]. Following this framework, a larger number of PGD-based AT methods are proposed, amongst which an early stopping version [27] stands out. The algorithm of PGD-AT variants is summarized in Algorithm 1. Fast Adversarial Training. Fast adversarial training variants are proposed recently by adopting the one-step fast gradient sign method (FGSM) [6], which are also dubbed FGSM-based AT. It can be defined as: end for 9: end for where represents the maximal perturbation strength. Although it accelerates adversarial training, it severely damages the robustness of the model, i.e., it cannot defend against the PGD attack [32], [33]. FGSM-based AT has a catastrophic overfitting issue, i.e., the target model suddenly loses the robustness accuracy of adversarial examples generated by PGD (on the training data) during training. Fortunately, Wong et al. [31] propose to use FGSM-based AT combined with a random initialization to relieve the catastrophic overfitting. It can achieve comparable performance to the prime PGD-based AT [12]. Specifically, they perform FGSM with a random initialization η ∈ U(− , ), where U represents a uniform distribution, which can be called FGSM-RS. It can be defined as: where α represents the step size, which is set to 1.25 in [31]. This work demonstrates that combined with a good initialization, FGSM-based AT can achieve excellent performance as PGD-AT [12]. More importantly, compared with PGD-AT, the FGSM-RS requires a lower computational cost. The FGSM-RS algorithm is summarized in Algorithm 2. Moreover, following FGSM-RS, several works are proposed to improve model robustness. Andriushchenko et al. [32] find using a random initialization does not completely solve the catastrophic overfitting issue and propose a regularization method, dubbed FGSM-GA, to improve the performance of FGSM-based AT. Moreover, Kim et al. [33] propose a stable single-step adversarial training based on FGSM-RS, a.k.a., FGSM-CKPT. FGSM-CKPT determines an appropriate magnitude of the perturbation for each image and thus prevents catastrophic overfitting.

III. THE PROPOSED METHOD
For fast AT, using a random sample-agnostic initialization is common and facilitates the efficiency, yet it hinders a further model robustness improvement. To remedy this issue, we propose a sample-dependent adversarial initialization to end for 7: end for improve the robustness of fast AT as well as to overcome the catastrophic overfitting issue. The pipeline of the proposed method is introduced in Sec. III-A, the architecture of the proposed generative network is introduced in Sec. III-B and the formulation is introduced in Sec. III-C.

A. Pipeline of the Proposed Method
The proposed method consists of two networks, i.e., a generative network and a target network. The former one learns to produce a dynamic sample-dependent adversarial initialization for FGSM to generate adversarial examples, instead of using a random initialization. And the latter adopts the generated adversarial examples for training to improve model robustness. As shown in Fig. 2, a benign image and its gradient information from the target network are fed to the generative network and the generative network generates a sample-dependent initialization. FGSM is then performed on the input image added with the generated initialization to generate adversarial examples. The target network is trained on the adversarial examples to improve the robustness against adversarial attacks.
For the target network, we adopt the architecture of a typical image classification network, defined as y = f (x; w), where x represents an input image, y represents the predicted label, and w represents the parameters of the target network.
The generative network consists of three layers. The detailed structure of the generative network is presented in Sec. III-B. The inputs of the generative network are the benign image and its signed gradient. The signed gradient can be calculated as: where x is the input image and y is the ground-truth label.
The initialization generation process can be defined as: where g(·; θ) represents the generative network with the parameters θ, and η g represents the generated adversarial initialization. The output pixel value space of g(·; θ) is is a scale factor that maps the value to the range of [− , ].

B. Architecture of the Generative Network
The architecture of the generative network is shown in Fig. 3. We combine the clean image with its gradient information from the target network to form the input of the generative network. The generative network generates a sample-dependent adversarial initialization. We adopt a lightweight generative network, which only includes three layers. The detailed setting of each layer is shown in Table I. The first layer consists of one convolutional layer with 64 filters of size 3 × 3 × 6 which is followed by a batch normalization layer [44]. The second layer is a ResBlock [45] with 64 filters of size 3 × 3 × 64. And the third layer consists of one convolutional layer with 64 filters of size 3 × 3 × 3 which is followed by a batch normalization layer. We adopt the ReLU [46] as an activation function.
The proposed generative network generates a sampledependent initialization for the clean image based on itself and its gradient information. Compared with the random initialization, the proposed sample-dependent adversarial initialization is more informative. A series of experiments demonstrate that the proposed FGSM-SDI not only overcomes the catastrophic overfitting but also reduces the gap between it and the multistep AT.

C. Formulation of the Proposed Method
Similar to the adversarial perturbation of FGSM-RS [31] in Eq. 4, our perturbation i.e., the approximate solution of the inner maximization problem, can be written as: where η g is the adversarial initialization defined in Eq. 6, generated by the generative network. Note that our perturbation involves the parameters θ of the generative network via the initialization. The distinctive difference between the perturbation of FGSM-RS (Eq. 4) and that of our FGSM-SDI (Eq. 7) lies in the initialization term η g . The initialization of FGSM-RS is randomly sampled without any guidance of other information.  Though it can improve the diversity of adversarial examples and the robustness, it encounters the catastrophic overfitting issue that the robustness drops sharply in the late training stage (see Fig. 5). Differently, our initialization η g is a function of the benign image x and its gradient information s x that provides some informative guidance on the direction of the initial perturbation. It not only overcomes the catastrophic overfitting issue but also greatly improves the robustness compared to current fast AT methods, even comparable to PGD-AT. Please refer to the results of the comparative experiment in Sec. IV-E. With the definition of our perturbation, the objective function of jointly learning the generative network and the target network can be derived as follows. From the objective function of standard AT in Eq. 1, our solution of the inner maximization problem involves the parameters of the generative network. When fixing the parameters θ, the solution is approximated by δ g in Eq. 7. We can further maximize the loss by searching for better parameters θ, i.e., max θ L(f (x + δ g (θ); w), y). Hence, the objective function of our joint optimization can be defined as: As viewed in Eq. 8, the generative network plays a game with the target network. The former maximizes the loss to generate an effective initialization for the adversarial example generation, while the latter minimizes the loss to update the parameters to gain model robustness against adversarial examples. More importantly, the generative network can generate initializations according to the robustness of the target model at different training stages. This minimax problem can be solved by alternatively optimizing w and θ. Note that we update θ and w iteratively. We update θ every k times of updating w. And k is a hyper-parameter that needs to be tuned. The algorithm if i mod k = 0 then 5: end for 13: end for for solving this problem is shown in Algorithm 3.
Eq. 1 is the objective of standard and fast AT methods. Compared to Eq. 1, our formulation has the following differences. First, δ in Eq. 1 is a variable to optimize, while we replace it with the approximate solution δ g (θ) of the inner maximization. δ g (θ) is regarded as a function that involves the parameters of the generative network. Second, we apply an additional maximization to the parameters of the generative network to further maximize the loss, which forces the competition between the two networks.
Connection to Two-step PGD-AT. In our adversarial example generation process (see Fig. 2), we calculate the gradient twice with FGSM, i.e., one as input of the generative network for initialization generation and the other for adversarial example generation. However, our method is quite different from the two-step PGD-AT method (PGD2-AT) with the number of iterations being 2. PGD2-AT can be regarded as a fast AT method that straightforwardly uses the gradient in the first step as initialization. Such initialization limits the diversity of adversarial examples as it is bounded by a fixed step size, a pre-defined projection range, and the sign operation (see Eq. 2). Our method uses a generative network to produce the initialization without the setting of step size or projection. The adversarial initialization provides a perturbation to the gradient, which enriches the diversity of adversarial examples and further improves model robustness. Experimental evaluations show the superiority of our method against PGD2-AT (see Table III).

IV. EXPERIMENTS
To evaluate the effectiveness of our FGSM-SDI, extensive experiments are conducted on four benchmark databases, including the selection of hyper-parameters in the proposed FGSM-SDI, the ablation study of the adversarial example generation, and the comparisons with state-of-the-art fast AT methods.

A. Experimental Settings
Datasets. We adopt four benchmark databases to conduct experiments, i.e., CIFAR10 [47], CIFAR100 [47], Tiny Ima-geNet [48] and ImageNet [48]. They are the most widely used databases to evaluate adversarial robustness. Both CIFAR10 and CIFAR100 consist of 50,000 training images and 10,000 test images. The image size is 32 × 32 × 3. CIFAR10 covers 10 classes while CIFAR100 covers 100 classes. Tiny ImageNet is a subset of the ImageNet database [48], which contains 200 classes. Each class has 600 images. The image size is 64 × 64 × 3. As for the ImageNet database, it contains 1000 classes and we resize the image to 224×224×3. Following the setting of [28], as Tiny ImageNet and ImageNet have no labels for the test dataset, we conduct evaluations on the validation dataset. Experimental Setups. On CIFAR10, ResNet18 [49] and WideResNet34-10 [50] are used as the target network. On CIFAR100, ResNet18 [49] is used as the target network. On Tiny ImageNet, PreActResNet18 [51] is used as the target network. On ImageNet, ResNet50 [49] is used as the target network. As for CIFAR10, CIFAR100, and Tiny ImageNet, following the settings of [27], [52], the target network is trained for 110 epochs. The learning rate decays with a factor of 0.1 at the 100-th and 105-th epoch. We adopt the SGD [53] momentum optimizer with an initial learning rate of 0.1 and the weight decay of 5e-4. As for ImageNet, following the previous study [30], [31], the target network is trained for 90 epochs. The learning rate decays with a factor of 0.1 at the 30-th and 60-th epoch. The SGD momentum optimizer is used with an initial learning rate of 0.1 and the weight decay of 5e-4. Note that we report the results of the checkpoint with the best accuracy under the attack of PGD-10 as well as the results of the last checkpoint. For adversarial robustness evaluation, Fig. 4. Robustness accuracy of the proposed FGSM-SDI with different interval k. We adopt Resnet18 on the CIFAR10 database to conduct experiments we adopt several adversarial attack methods to attack the target network, including FGSM [6], PGD [12], C&W [35], and AA [41]. And the maximum perturbation strength is set to 8/255 for all attack methods. Moreover, we conduct the PGD attack with 10, 20, and 50 iterations, i.e., PGD-10, PGD-20, and PGD-50. We run all our experiments on a single NVIDIA Tesla V100 based on which the training time is calculated. We also conduct comparison experiments using a cyclic learning rate strategy [54].

B. Hyper-parameter Selection
There is one hyper-parameter in the proposed FGSM-SDI, i.e, the interval k. We update θ every k times of updating w. This hyper-parameter not only affects model training efficiency but also affects model robustness against adversarial examples. To select the optimal hyper-parameter, we conduct a hyperparameter selection experiment on CIFAR10. The results are shown in Fig. 4. The calculation time of the proposed FGSM-SDI decreases along with the increase of parameter k. That is, the smaller the k value is, the more frequently the generative network is updated, then the generative network requires more calculation time for training. Surprisingly, when k = 1 ∼ 20, the performance against adversarial examples improves with the increase of parameter k. When k = 20 ∼ 40, the performance against adversarial examples slightly drops with the increase of parameter k. When k = 20, the proposed FGSM-SDI achieves the best adversarial robustness in all adversarial attack scenarios. Considering adversarial training efficiency, we set k to 20.

C. Ablation Study
As shown in Fig. 2, the generative network takes a benign image and its signed gradient as input to produce an initialization for FGSM to generate adversarial examples. Here, we study the influence of each input on the quality of the initialization. Moreover, as mentioned in Sec. III-C, since our method has a certain connection to the two-step PGD-AT, we also conduct experiments to compare with it.
The results of studying the inputs of the generative network are shown in Table II. 'Benign' represents that only the benign image is fed into the generative network, while 'Grad'  represents that the signed gradient is the only input. 'Benign + Grad' represents that both the benign image and signed gradient are regarded as input. Analyses are summarized as follows. First, it can be observed that Benign + Grad achieves the best performance in robustness under all attacks by exploiting both types of inputs, which indicates that each input contributes to the final performance. Second, the gradient information plays a more important role than the benign image because Grad outperforms Benign by a large margin, especially under the attack methods in AA. Third, only using a benign image as input cannot produce a desirable initialization and still suffers from the catastrophic overfitting issue, as the robustness of Benign drops dramatically in the late training phase, i.e., the performance of the last checkpoint is much worse than that of the best checkpoint.
The comparison with the two-step PGD-AT (i.e., PGD2-AT [27] ) is shown in Table III. Following the setting of [32], [33], we use the step size α = /2 to conduct PGD2-AT. Both PGD2-AT and our FGSM-SDI calculate the gradient w.r.t input sample twice by FGSM to generate adversarial examples. Their differences are stated in Sec. III-C. It can be observed that our FGSM-SDI can achieve much better performance in robustness than PGD2-AT in all the attack scenarios. For example, compared with PGD2-AT, our FGSM-SDI improves the performance under the strong attack (AA) by about 4% on the best and last checkpoints. PGD2-AT can be viewed as a method that uses the gradient of the first FGSM as initialization for the second FGSM. Since the initialization is exactly determined by the gradient, it limits the diversity of subsequently generated adversarial examples. We alleviate this issue by introducing a generative network to learn how to generate initialization under the game mechanism. The superiority of our method demonstrates the feasibility of this strategy. Compared with PGD2-AT, our FGSM-SDI costs a little more calculating time, but it achieves much higher robustness under all the attack scenarios. To further verify the effectiveness of our FGSM-SDI, we compare our FGSM-SDI with the four PGD-AT (i.e., PGD4-AT [27] ). Following the setting of [55], we use the step size α = /4 to conduct PGD4-AT. The result is shown in Table III. Compared with PGD4-AT, our FGSM-SDI not only achieves much better robustness performance in all the attack scenarios but also costs less calculating time for training. Specifically, as for the strong attack (AA), our FGSM-SDI improves the performance by about 3% on the best and last checkpoints. As for the training efficiency, our FGSM-SDI reduces the training time by about 36 minutes. Note that there is a trade-off between the clean accuracy and the robustness, better robustness always comes along with a lower clean accuracy [26]. Although our FGSM-SDI reduces the clean accuracy slightly, it improves the robust accuracy greatly.

D. Relieving Catastrophic Overfitting
Catastrophic overfitting [31]- [33] is one of the tough problems that mostly affect the model robustness of the fast AT methods, which refers to the phenomenon that the accuracy on  Fig. 5. The accuracy of FGSM-RS decreases rapidly to 0 after about 70-th epoch. Although other fast AT methods overcome the catastrophic overfitting issue, their performance is far from satisfactory, i.e., there is a huge gap between the fast AT methods and the advanced multi-step PGD-AT [27]. For example, FGSM-GA achieves the accuracy of 58.46% and 48.17% on training and test data respectively, while PGD-AT achieves the much higher accuracy of 63.71% and 53.33%. Differently, our FGSM-SDI can not only overcome the catastrophic overfitting but also achieve comparable robustness to PGD-AT. Specifically, it achieves the accuracy of 64.14% and 52.81% on training and test data, respectively. Note that our FGSM-SDI costs only one-third the computation time of PGD-AT and also less time than FGSM-GA. More details will be discussed in the following part.

E. Comparisons with State-of-the-art Methods
We compare our FGSM-SDI with several state-of-the-art fast AT methods (i.e., Free [30], FGSM-RS [31], FGSM-GA [43], and FGSM-CKPT [33]) and an advanced multi-step AT method ( i.e., PGD-AT [27]) which adopts 10 steps to generate adversarial examples on four benchmark databases. We follow the settings reported in their original works to train these AT methods. Note that to ensure fairness of comparison, we do not divide the number of epochs by m such that the total number of epochs remains the same as the other fast AT methods. Results on CIFAR10. We adopt Resnet18 as the target network to conduct the comparison experiment with other defense methods on CIFAR10. The results are shown in Table IV. Compared with the fast AT methods, our method achieves the best performance under all attack scenarios and comparable robustness to the advanced PGD-AT [27]. The previous most fast AT methods are only as effective as the prime PGD-AT [12], i.e., they achieve the performance of about 45% under the PGD-10 attack. The performance is far from that of the advanced PGD-AT [27] which uses an early stopping trick to achieve above 50% accuracy. Unlike them, our method can achieve more than 53% under the PGD-10 attack on the best and last checkpoint. As for the strong attack (AA), the previous most powerful fast AT method (FGSM-GA) achieves the performance of about 43%, but the proposed FGSM-SDI     achieves about 48% robust accuracy which is the same as PGD-AT. In terms of training efficiency, our training time is less than FGSM-GA, Free, and PGD-AT. Specifically, the training time of our FGSM-SDI is about 1.6 times the training time of FGSM-RS [31]. Though FGSM-RS and FGSM-CKPT are more efficient than our method, their performance is always the worst among all the fast AT methods. FGSM-RS is the fastest method that uses a random initialization. Our method improves the initialization to boost the robustness by introducing the generative network, resulting in the sacrifice of efficiency for an additional gradient calculation. Therefore, our method can be viewed as a method that balances the robustness and the training efficiency. Moreover, we adopt WideResNet34-10 which is a large architecture model to conduct a comparison experiment. The results are shown in Table V. We observe a similar phenomenon as the Resnet18 trained on CIFAR10. Our FGSM-SDI achieves the best performance under all attack scenarios compared with previous fast AT methods. Besides, compared with the advanced PGD-AT, the proposed FGSM-SDI costs less time and achieves comparable robustness to it. Specifically, PGD-AT achieves the performance of about 51% and 49% against AA attack on the best and last checkpoints. Our FGSM-SDI also achieves the performance of about 51% and 49%. But PGD-AT takes about 1914 minutes for training, while our FGSM-SDI only takes about 533 minutes for training. We also conduct comparative experiments using a cyclic learning rate strategy [54] on CIFAR10. Following [32], [33],  we set the maximal learning rate of FGSM-GA [32] and FGSM-CKPT [33] to 0.3. Following [31], we set the maximal learning rate of FGSM-RS [31], Free [30], PGD-AT [27], and the proposed method to 0.2. We train all the models for 30 epochs. Other training and evaluation settings remain unchanged. The results are shown in Table VI. Compared with the other fast AT methods, the proposed FGSM-SDI achieves the best adversarial robustness and comparable robustness to the advanced PGD-AT [27]. Using a cyclic learning rate strategy can prevent catastrophic overfitting for the fast AT methods, i.e., the performance of the last checkpoint is almost the same as that of the best checkpoint. But their adversarial robustness is still far from that of the advanced PGD-AT [27]. Differently, our FGSM-SDI can achieve comparable robustness to PGD-AT [27]. For example, FGSM-RS [31] achieves about 42% accuracy under AA attack, while our method achieves about 46%. In terms of efficiency, our method outperforms Free and FGSM-GA and is much better than PGD-AT. FGSM-RS and FGSM-CKPT use a random initialization which promotes efficiency at a sacrifice in robustness. Our method improves the initialization with a generative network, which greatly boosts the adversarial robustness with the slight expense of time cost.
Results on CIFAR100. The results are shown in Table VII. The CIFAR100 database covers more classes than the CIFAR10, which makes the target network harder to obtain robustness. We can observe a similar phenomenon as on CIFAR10. In detail, compared with the other fast AT methods, our FGSM-SDI achieves the best adversarial robustness under all adversarial attack scenarios. For example, the previous fast AT methods achieve the performance of about 20% under the PGD-50 attack which is far from that of the advanced PGD-AT [27] which achieves about 28% accuracy. While the proposed FGSM-SDI achieves the performance of about 30% under the PGD-50 attack. Surprisingly, our method can even outperform PGD-AT under the attacks of PGD-10, PGD-20, PGD-50, and APGD. Our method also achieves comparable robustness to the advanced PGD-AT under the strong attack methods (C&W and AA). And our clean accuracy is also about 3% higher than PGD-AT [27]. This indicates the potential of our method in boosting robustness. In terms of training efficiency, similar results are observed on CIFAR10. Our FGSM-SDI can be 3 times faster than the advanced PGD-AT [27]. Although our FGSM-SDI costs a little more time than FGSM-RS, it not only relieves the catastrophic overfitting problem but also achieves   comparable robustness to the advanced PGD-AT.
Results on Tiny ImageNet. The results are shown in Table  VIII. Tiny ImageNet is a larger database compared to CIFAR10 and CIFAR100. Performing AT on Tiny ImageNet requires more computational cost. It takes about 1,833 minutes for PGD-AT to conduct training. But, our FGSM-SDI only takes 565 minutes and achieves comparable or even better robustness than PGD-AT. Similar to the results on CIFAR100 and CIFAR10, compared with previous fast AT methods, our FGSM-SDI achieves the best performance under all attack scenarios. Moreover, compared with the advanced PGD-AT, our FGSM-SDI achieves better performance under all attack scenarios even the strong attack (AA). Specifically, PGD-AT achieves the performance of about 16% and 13% accuracy under AA attack on the best and last checkpoints, while our FGSM-SDI achieves the performance of about 17% and 14% accuracy. Moreover, FGSM-SDI achieves higher clean accuracy compared with PGD-AT. Specifically, our clean accuracy is also about 3% higher than PGD-AT. The efficiency comparison is similar to that on CIFAR10 and CIFAR100.
Results on ImageNet. Following [30], [31], we adopt Resnet50 to conduct AT on ImageNet under the maximum perturbation strength = 2, = 4, and = 8. The results are shown in Table IX. When = 2, all methods achieve roughly the same robustness against adversarial examples. But as the maximal perturbation strength becomes larger, PGD-AT and our FGSM-SDI achieves better robustness performance. Especially, when = 8, the FGSM-RS cannot defend against the PGD-based attacks. But our FGSM-SDI still achieves the performance of about 26% and 21% under the PGD-10 and PGD-50 attacks and achieves comparable robustness to PGD-AT. In terms of training efficiency, similar phenomenons are observed on other databases, our FGSM-SDI can be 3 times faster than the advanced PGD-AT. They do not meet the catastrophic overfitting. This observation indicates that the catastrophic overfitting is associated with the adversarial example quality in the training process. Moreover, the attack success rate of adversarial in the training process is also related to the robust performance. The PGD-AT that adopts the adversarial examples with the highest attack success rate has the best robust performance. Compared with PGD2-AT, our FGSM-SDI has a higher attack success rate and achieves a better robust performance.
The generative network is one of the core parts of the proposed method. We adopt ResNet18 as the target model on CIFAR10 to explore the impact of the generative network. In detail, when training the generator, we perform m iterations on it, which can be dubbed FGSM-SDI-m. We record the robustness accuracy of adversarial examples generated by PGD-10 on test data in the training phase. The robustness accuracy curves under the PGD-10 attack are shown in Fig. 7. It can be observed that improving the training iteration of the generator can improve the robustness performance, especially at the beginning of training. That indicates that model robustness increases as generator training progresses. And we also visualize the adversarial initialization and FGSMupdated perturbations for the FGSM-RS and our FGSM-SDI among continuous training epochs. As shown in Fig. 8, it can be observed that compared with the random initialization, the proposed initialization is more informative.
Adversarial perturbations fool a well-trained model by interfering with important local regions that determine image classification. To explore whether our FGSM-SDI will be affected by adversarial perturbations, we adopt Gradientweighted Class Activation Mapping (Grad-CAM) [56] to generate the heat maps that locate the category-related areas in the image. As shown in Fig. 9, it can be observed that as for FGSM-RS, adversarial perturbations modify the distribution of the maximal points on the generated heat map, while as for our FGSM-SDI, the adversarial perturbations do not modify the distribution of the maximal points on the generated heat-map. That indicates that our FGSM-SDI is more robust. Moreover, we compare the loss landscape of the proposed method with those of the other fast AT methods to explore the association between latent hidden perturbation and local linearity. As shown in Fig 10, compared with other AT methods, the cross-entropy loss of our FGSM-SDI is more linear in the adversarial direction. Using the latent perturbation generated by the proposed method can preserve the local linearity of the target model better. It qualitatively proves that using the proposed sample-dependent adversarial initialization can boost the fast AT. And to explore the stability of the proposed method FGSM-SDI, we train the proposed method multiple times and record the robustness accuracy of adversarial examples generated by PGD-10 on test data in the training phase. The mean and variance of robustness accuracy is shown in Fig 11. It can be observed that the proposed method keeps stable robustness accuracy against adversarial examples.

V. CONCLUSION
In this paper, we propose a sample-dependent adversarial initialization to boost fast AT. Specifically, we adopt a generative network conditioned on a benign image and its gradient information from the target network to generate an effective initialization. In the training phase, the generative network and the target network are optimized jointly and play a game. The former learns to produce a dynamic sample-dependent initialization to generate stronger adversarial examples based on the current target network. And the latter adopts the generated adversarial examples for training to improve model robustness. Compared with widely adopted random initialization fashions in fast AT, the proposed initialization overcomes the catastrophic overfitting, thus improves model robustness. Extensive experimental results demonstrate the superiority of our proposed method.