{'fraction_non_alphanumeric': 0.03148002281554717, 'fraction_numerical': 0.01461281473232474, 'mean_word_length': 5.028819387588014, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Understanding Catastrophic Overfitting in Single-step Adversarial Training

Adversarial examples are perturbed inputs that are designed to deceive machine-learning classifiers by adding adversarial perturbations to the original data. Although fast adversarial training have demonstrated both robustness and efficiency, the problem of"catastrophic overfitting"has been observed. It is a phenomenon that, during single-step adversarial training, the robust accuracy against projected gradient descent (PGD) suddenly decreases to 0% after few epochs, whereas the robustness against fast gradient sign method (FGSM) increases to 100%. In this paper, we address three main topics. (i) We demonstrate that catastrophic overfitting occurs in single-step adversarial training because it trains adversarial images with maximum perturbation only, not all adversarial examples in the adversarial direction, which leads to a distorted decision boundary and a highly curved loss surface. (ii) We experimentally prove this phenomenon by proposing a simple method using checkpoints. This method not only prevents catastrophic overfitting, but also overrides the belief that single-step adversarial training is hard to prevent multi-step attacks. (iii) We compare the performance of the proposed method to that obtained in recent works and demonstrate that it provides sufficient robustness to different attacks even after hundreds of training epochs in less time. All code for reproducing the experiments in this paper are at https://github.com/Harry24k/catastrophic-overfitting.


Introduction
Adversarial examples are perturbed inputs that are designed to deceive machine learning classifiers by adding adversarial noises to the original data. The perturbations are sufficiently subtle such that a human can not notice the difference, yet the classifier produces an incorrect classification. Since deep-learning models were found to be vulnerable to adversarial examples (Szegedy et al. 2013), a line of work was proposed to mitigate the problem and improve the robustness of the models. Among the numerous defensive methods, projected gradient descent (PGD) adversarial training (Madry et al. 2017) is one of the most successful approaches to make models robust against adversarial attacks. Although PGD adversarial training serves as a strong defensive algorithm, as it relies on a multi-step adversarial attack, high computation cost is required for multiple forward and back propagation during batch training; for example, it could require several days to train a large model.
To overcome this issue, other studies (Shafahi et al. 2019;Wong, Rice, and Kolter 2020) that attempt to reduce the computational burden of adversarial training using single-step adversarial attacks (Goodfellow, Shlens, and Szegedy 2014) have been proposed. Wong, Rice, and Kolter (2020) suggested fast adversarial training, which is a modified version of fast gradient sign method (FGSM) adversarial training designed to be as effective as PGD adversarial training. Fast adversarial training have demonstrated both robustness and efficiency; however, it suffers from the problem of "catastrophic overfitting", which is a phenomenon where the robust accuracy against PGD suddenly decreases to 0%, whereas the robustness against FGSM increases rapidly to near 100%. Wong, Rice, and Kolter (2020) first discovered this issue and used early stopping to prevent it. Later, it was found that catastrophic overfitting also occurs in different single-step adversarial training models such as Free adversarial training (Andriushchenko and Flammarion 2020).
In this regard, few attempts have been made to discover the underlying reason for catastrophic overfitting and methods proposed to prevent this failure. Andriushchenko and Flammarion (2020) proposed use of gradient alignment. Concurrently, Li et al. (2020) proposed using PGD adversarial training temporally when the PGD robustness decreases rapidly. However, these approaches were computationally inefficient or did not provide a fundamental solution to the problem. In this paper, we first study the underlying reason for catastrophic overfitting. We analyze the differences before and after catastrophic overfitting and determine the reasons why this failure occurs in single-step adversarial training. Unlike the previous notion that a larger perturbation implies a stronger attack, we observed the following phenomenon during adversarial training: sometimes, a smaller perturbation deceives the model, whereas the model is robust against larger perturbations. We call this phenomenon "decision boundary distortion". Figure 1 presents an example of decision boundary distortion by visualizing the loss surface. The origin indicates the original image x whose label is "dog". v 1 is a direction of a single-step adversarial perturbation and v 2 is a random direction. The adversarial image x + v 1 is classified as the correct label, but there is a distorted interval where x + k · v 1 is misclassified even k is less than 1, which indicates that the model is only robust to the attack when the magnitude of the perturbation exactly equals the maximum perturbation , and not to other perturbations that are weaker. Considering multi-step attacks can explore more areas due to the small step size, it becomes an easy task to find the perturbation in a distorted interval. Thus, when the decision boundary distortion occurs, the model becomes much robust against single-step attack, but reveals fatal weaknesses to multi-step attacks and leads to catastrophic overfitting.
We identify the relationship between the distortion of the decision boundary and catastrophic overfitting. We empirically discover the causal relationship between the decision boundary distortion and vulnerability against multi-step attacks by defining and measuring a new metric. This metric, called distortion, is designed to check how many examples have a distorted interval for a given model. Furthermore, we present a simple algorithm that determine the appropriate magnitude of the step size for each image that should be used in adversarial training and show that the proposed method successfully avoids catastrophic overfitting. We find that the single-step adversarial training can give sufficient robustness to the model against multi-step attacks by observing training robust accuracy, which implies the problem of single-step adversarial training is the decision boundary distortion, not the direction of the attack.

Contribution:
• We found a phenomenon "decision boundary distortion" that occurs during single-step adversarial training.
Then we empirically discovered the underlying connection between the distortion and the catastrophic overfitting.
• We suggested a simple method that can prevent decision boundary distortion by searching the appropriate step size for each image during adversarial training. This method not only prevents catastrophic overfitting, but also achieves 100% accuracy for the training examples against PGD.
• We evaluated the robustness of our method against various adversarial attacks including FGSM, PGD, and AutoAttack (Croce and Hein 2020) and demonstrated the proposed method can provide sufficient robustness without catastrophic overfitting.

Adversarial Robustness
There are two major movements for building a robust model: provable defenses and adversarial training.
A considerable amount of literature related to the certified defenses of deep-learning models has been published. Provable defenses attempt to provide provable guarantees for robust performance, such as linear relaxations (Wong and Kolter 2018;Zhang et al. 2019a), interval bound propagation (Gowal et al. 2018), and randomized smoothing (Cohen, Rosenfeld, and Kolter 2019). However, provable defenses are computationally inefficient compared to adversarial training. The empirical performance of provable defenses under conventional adversarial attacks is less robust and accurate than that of adversarial training.
Adversarial training is an approach that augments the adversarial examples generated by adversarial attacks (Goodfellow, Shlens, and Szegedy 2014;Madry et al. 2017;Tramèr et al. 2017). Because this approach is simple and achieves high empirical robustness for different attacks, it is widely used and developed along with other deep learning methods such as mix-up (Lamb et al. 2019;Pang, Xu, and Zhu 2019) and unsupervised training (Uesato et al. 2019;Najafi et al. 2019;Carmon et al. 2019).
In this paper, we focus on adversarial training. Given an example (x, y) ∼ D, let (x, y; θ) = (f θ (x), y) denotes the loss function of a deep learning model f with parameters θ. Then, adversarial training with a maximum perturbation can be formalized as follows: min A perturbation δ is in B(x, ) which denotes -ball around an example x with a specific distance measure. The most-used distance measures are L 0 , L 2 or L ∞ . In this paper, we use L ∞ as the distance measure.
However, the above optimization is considered as an NP-hard problem because it contains a non-convex min-max problem. Thus, instead of the inner maximization problem, an adversarial attack is used to gather the perturbed images.
Fast Gradient Sign Method (FGSM). FGSM (Goodfellow, Shlens, and Szegedy 2014), the simplest adversarial attack, computes a gradient to increase the loss and uses a sign of the gradient to find an adversarial image x . Because FGSM requires only one gradient, it is considered as the least expensive adversarial attack (Goodfellow, Shlens, and Szegedy 2014;Madry et al. 2017).
Projected Gradient Descent (PGD). On the other hand, PGD (Madry et al. 2017) uses multiple gradients to generate more powerful adversarial images. With a step size α, PGD can be formalized as follows: where Π B(x, ) refers the projection to the -ball B(x, ). We note that PGDn corresponds to PGD with n iterations (or steps). For instance, PGD7 denotes the number of PGD iterations is 7.

Single-step attack vs. Multi-step attack
Single-step adversarial training was previously believed to be a non-robust method because it produces nearly 0% accuracy against PGD (Madry et al. 2017). Moreover, a model trained with FGSM has been confirmed to have typical characteristics, such as gradient masking, which indicates that a single-step gradient is not sufficient to find a decent adversarial example owing to the highly curved loss surface (Tramèr et al. 2017). For the above reasons, a number of studies have been conducted on multi-step attacks.
Contrary to this perception, however, free adversarial training (Shafahi et al. 2019) has achieved remarkable performance with a single-step gradient using redundant batches and accumulative perturbation. Following Shafahi et al. (2019), Wong, Rice, and Kolter (2020) proposed fast adversarial training using FGSM with an uniform random initialization. Fast adversarial training shows almost equivalent performance to PGD (Madry et al. 2017) and Free adversarial training (Shafahi et al. 2019).

Catastrophic Overfitting
Although fast adversarial training is fast and performs well, a previously undiscovered phenomenon was identified. During single-step adversarial training, there is a singular point at which the robustness of the model against PGD falls sharply. This phenomenon is called catastrophic overfitting. Fast adversarial training (Wong, Rice, and Kolter 2020) temporally avoids catastrophic overfitting by tracking the robustness accuracy against PGD on the training batches and using early stopping.
However, to perform early stopping, one must continue checking the robustness against PGD; furthermore, the standard accuracy does not yield the maximum potential (Andriushchenko and Flammarion 2020). To solve these shortcomings and gain a deeper understanding of catastrophic overfitting, a line of work has been proposed. Babu et al. (2020) identified that catastrophic overfitting arises with early overfitting to FGSM. To prevent this type of overfitting, they introduced dropout scheduling and demonstrated stable adversarial training up to 100 epochs. Conversely, Li et al. (2020) trained the model with FGSM in the first phase and then changed to PGD when there was a large decrease in the PGD accuracy. Andriushchenko and Flammarion (2020) found that an abnormal behavior of a single filter leads to nonlinear locality with single-layer convolutional networks. Based on this observation, they measured cos(∇ x , ∇ x+η ), the gradient alignment inside the perturbation set, and proposed a regularization method, GradAlign, which prevents catastrophic overfitting by inducing the gradient alignment.
Even with an increased understanding of catastrophic overfitting and methods to prevent it, the key question remains unanswered: What characteristic of single-step attacks is the cause of catastrophic overfitting?
In this paper, we discuss the cause of catastrophic overfitting in the context of single-step adversarial training. We also propose a new simple method to facilitate stable single-step adversarial training, wherein longer training can produce higher standard accuracy with sufficient adversarial robustness.

Revisiting Catastrophic Overfitting
First, to analyze catastrophic overfitting, we start by recording the robust accuracy of fast adversarial training on CIFAR-10 (Krizhevsky, Hinton et al. 2009). The maximum perturbation is fixed to 8/255. We use FGSM and PGD7 to verify the robust accuracy with the same settings = 8/255 and 2/255 as a step size α. In plot (a), after 240 batches, the robustness against PGD7 begins to decrease rapidly; conversely, the robustness for FGSM increases. Plot (b) indicates the mean of absolute value of PGD7 perturbation E[||δ P GD7 || 1 ] and squared L 2 norm of the gradient of the images E[||∇ x || 2 ] corresponding to the image above. After catastrophic overfitting, there is a trend of decreasing the mean perturbation. This is consistent with the phenomenon wherein the perturbations of the catastrophic overfitted model are located away from the maximum perturbation unlike the early stopped model (Wong, Rice, and Kolter 2020). Concurrently, a significant increase in the squared L 2 norm of the gradient is also observed. The highest point indicates a large difference, approximately 35 times greater than that before catastrophic overfitting.

Decision Boundary Distortion Catastrophic Overfitting
Original image Robust against FGSM Figure 3: The process of normal decision boundary turns into distorted decision boundary. The left figure shows the loss surface before catastrophic overfitting with a FGSM adversarial direction v 1 and a random direction v 2 . The red point denotes an adversarial example x + v 1 generated from the original image x whose label is "dog". Figure in the middle shows the changed loss surface right after learning the adversarial example x + v 1 . Here, v 1 is the same vector as the one in the left figure. The distorted interval begins to occur for the first time in this middle figure. As training continues, the distorted decision boundary grows uncontrollably so that the robustness against multi-step attacks decreases.
These two observations, the low magnitude of perturbation and the high gradient norm of the loss, make us wonder what would the loss surface actually looks like. Figure 3 illustrates the progress of adversarial training where catastrophic overfitting occurs. It displays the loss surface of the perturbed example where the green spot denotes the original images; the v 1 axis indicates the direction of FGSM, whereas the v 2 axis is a random direction. The true label of the original sample is "dog"; hence, the purple area indicates where the perturbed sample is correctly classified, whereas the blue area indicates the misclassified area. In the left figure, the red spot denotes the adversarial example that was used for adversarial training at the batch of the one in the left figure.
However, during fast adversarial training, the decision boundary becomes distorted, as indicated in the middle. In this figure, we can easily see that the model becomes more robust against FGSM, yet there exists a "distorted interval" in the FGSM direction. We call this phenomenon "decision boundary distortion". Contrary to the conventional belief that the larger the magnitude of the attack, the stronger the attack will be, there exists an interval that deviates from the decision boundary within the maximum epsilon. Considering the single-step adversarial training only trains a single adversarial example x , it is obvious that the model becomes strong against x . However, it is not clear that the model also becomes robust against images between x and x . In this case, the existence of a distorted interval makes the model vulnerable to multi-step attack.
To check the degree of decision boundary distortion, we defined the ratio of images in the distorted interval as 'distortion'. More precisely, given a deep learning model f and a loss function , the distortion d can be formalized as follows: where (x, y) ∼ D is an example and is the maximum perturbation. However, because the loss function of the model is not known explicitly, we use a number of samples to estimate S D and the distortion d.
In all experiments, we tested 100 samples in the perturbation direction δ = · sgn(∇ x ) for each example, i.e., k ∈ {0, 0.01, · · · , 0.99, 1}. Figure 2 (a), the distortion increases as catastrophic overfitting arises. This figure matches the observation in Figure 2 (b). When the robustness against PGD7 sharply decreases to 0%, the mean of absolute value of PGD7 perturbation E[||δ P GD7 || 1 ] decreases. It indicates that after catastrophic overfitting, the model reveals its weakness when the magnitude of the perturbation is small since the multi-step attack is capable to search the vulnerable region further inside B(x, ).

As in
The right figure of Figure 3 displays the loss surface after catastrophic overfitting. The model becomes considerably more robust against FGSM; however, the area of the distorted interval increases. It is easy to observe that the model is now perfectly overfitted for FGSM, yet loses its general robustness to other perturbations. Considering a directional derivative (∇ x ) T δ, in the process of having the same distorted decision boundary as the right figure, the loss surface inevitably becomes nonlinear and this indicates the norm of gradient of the loss increases, which matches with the To verify that decision boundary distortion is generally related to catastrophic overfitting, we demonstrated how the rate of decision boundary distortion and robustness to PGD7 attacks change in the batch during training (Figure 4). We illustrated the results on five different models: fast adversarial training (Fast Adv.) (Wong, Rice, and Kolter 2020), PGD2 (PGD2 Adv.) (Madry et al. 2017), TRADES (Zhang et al. 2019b), GradAlign (Andriushchenko and Flammarion 2020), and the proposed method (Ours). All the experiments were tested on setting as = 8/255 while α = 1.25 , α = 1/2 , and α = 1/4 for Fast Adv., PGD2 Adv., and TRADES, respectively. Experiments were also conducted on PGD adversarial training with different number of steps such as PGD4 and PGD7; however, because these showed similar results to PGD2 Adv., we only included PGD2 Adv. results. TRADES was trained with seven steps.
The key observation in Figure 4 is that the point where the decision boundary distortion begins in Fast Adv. (epoch 22) is identical to the point where robustness against PGD7 sharply decreases, that is, catastrophic overfitting occurs. Then, when decision boundary distortion disappears in Fast Adv. (epoch 45 to 72), the model immediately recovers the robust accuracy. Hence, we can assume that decision boundary distortion has underlying connection with the model's vulnerability against multi-step attacks. After epoch 72, the model once again suffers a catastrophic overfitting and never regains its robustness.

Stable Single-Step Adversarial Training
Based on the results in Section 3, we conclude that distorted decision boundary leads to catastrophic overfitting. We stress that the major cause of distorted decision boundary is that the point with a fixed distance from the original image is selected as an adversarial image instead of an optimal solution of inner maximum in Equation (1).
Previous single-step adversarial training searches for the best perturbation based on the linearity assumption. Under the linearity assumption, the most powerful adversarial perturbation δ would be same as · sgn(∇ x ) where is the maximum perturbation, and the following formula should be satisfied.
However, as confirmed in the previous session, the distortion of decision boundary with a highly curved loss surface has been observed in the training phase, which indicates is no longer the strongest adversarial step size in the direction of δ. We observed the high non-linearity of the loss function when catastrophic overfitting occurs. Now, we conclude that the main problem of single-step attacks is the failure of the linear approximation of inner maximization. Based on this assumption, we suggest a simple fix to prevent catastrophic overfitting by forcing the model to verify the inner interval of the adversarial direction. In this case, an appropriate magnitude of the perturbation should be taken into re-consideration instead of using : Algorithm 1: Stable single-step adversarial training Parameter :B mini-batches, a maximum perturbation , a step size α, and c check points for a network f θ .
Here, we introduce k that denotes the scaling parameter for the original adversarial direction sgn(∇ x ). In contrast to previous single-step adversarial training which uses fixed size of k = 1, we suggest to use an appropriate scaling parameter k * that can generate the stronger adversarial examples with a smaller perturbation as follows: In this way, regardless of linearity assumption, we can detect the distorted decision boundary by inspecting the inside of the distorted interval as shown in Figure 3. Simultaneously, we can train the model with adversarial examples that induces wrong classification in the direction of FGSM.
However, because we do not know the explicit loss function of the model, forward propagation is the only approach to check the adversarial images in the single-step attack direction. Hence, we propose a simple method to prevent distorted decision boundary. First, we calculate the single-step adversarial direction δ. Then, we choose multiple checkpoints x + 1 c δ, · · · , x + c−1 c δ, x + δ to verify whether there exists a distortion of the model. Here c denotes the number of checkpoints except for the clean image x, which was tested during the single-step attack process in advance. Then we input all checkpoints to the model and verify that the predicted labelŷ j matches the correct label y for all the checkpoints x + j c δ where j ∈ {1, · · · , c}. Among the incorrect images, including the clean image x, the smallest j is selected to be used in the training session; if all are correct, the adversarial image x = x + δ is used. Algorithm 1 is a summary of the proposed method.
Suppose the model has L layers with n neurons. Then, the time complexity of forward propagation is O(Ln 2 ). Considering that backward propagation has the same time complexity, generation of one adversarial example requires O(2Ln 2 ) in total. Thus, with c checkpoints, the proposed method consumes O((c + 4)Ln 2 ) because it requires one adversarial direction O(2Ln 2 ), forward propagation for c checkpoints O(cLn 2 ), and one optimization step O(2Ln 2 ). Compared to PGD2 adversarial training, which demands O(6Ln 2 ), the proposed method requires more time when c > 2 mathematically. However, the proposed method does not require additional memory when computing the predicted labels of the checkpoints because we do not need to track a history of variables for backward propagation; hence, larger validation batch sizes can be considered. Indeed, in Section 5, we observed the empirical results that the proposed method consumes less time than PGD2 adversarial training under c ≤ 4. Figure 4 illustrates that the proposed method can successfully avoid catastrophic overfitting, unlike fast adversarial training. Furthermore, the proposed model not only achieves nearly 100% robustness against PGD7, which fast adversarial training cannot accomplish, but also possesses the characteristics of zero distortion until the end of the training. This is the opposite to the common understanding that single-step adversarial training methods cannot perfectly defend against multi-step attacks.
Similarly, GradAlign also achieves high PGD7 accuracy for training set. Previously, Andriushchenko and Flammarion (2020) argued ∇ x and ∇ x+η tend to be perpendicular in catastrophic overfitted models where η is drawn from Uniform( , ). Considering the result of Figure 3, catastrophic overfitted models have a highly curved loss surface, which implies (∇ x ) T (∇ x+η ) is almost zero in high dimensions. Thus, the reason why GradAlign (Andriushchenko and Flammarion 2020) has the potential to avoid catastrophic overfitting is that the gradient alignment leads the model to learn a linear loss surface which reduces a chance of having distorted decision boundary. However, as illustrated in Figure 5, GradAlign failed to avoid catastrophic overfitting on a large dataset even though it requires significant computational overhead as much as PGD10, which indicates the gradient alignment is not the fundamental treatment.  In Figure 5, we performed the same experiment and observed robust accuracy and distortion on Tiny ImageNet. PGD2 adversarial training shows zero distortion during the training time and achieves nearly 100% PGD7 accuracy. Among single-step adversarial training methods, only the proposed method avoids catastrophic overfitting and achieves high PGD7 accuracy like multi-step adversarial training.
Compared to CIFAR10, as mentioned above, catastrophic overfitting occurs in GradAlign at 67th epoch. Considering Tiny ImageNet is a more complicated dataset, we can conclude that the proposed method provides more efficient and stronger algorithm than GradAlign to prevent catastrophic overfitting. This is also supported by the observation at 89th epoch. At this epoch, the distortion of GradAlign drops to 0% even PGD7 accuracy is nearly zero. Figure 6 illustrates that, although gradient alignment is working properly after catastrophic overfitting, the model can not escape from the local minima so that PGD7 accuracy is still zero.
The proposed model learns the image with the smallest perturbation among the incorrect adversarial images. In the initial states, the model outputs wrong predictions for almost every images so that min({k|ŷ i,k = y i } ∪ {1}) = 0 in Algorithm 1. As additional batches are trained, the average maximum perturbation E[||δ|| ∞ ] increases as in Figure  7 where δ = x − x and x is selected by the proposed method. To this end, one might think that our model is just a simple variation of -scheduling.
In order to point out the difference, fast adversarial training with -scheduling is also considered. For each epoch, we use the average maximum perturbation E[||δ|| ∞ ] calculated from the proposed method as the maximum perturbation . We summarized the results in Figure 7.
Notably, -scheduling could not help fast adversarial training to avoid catastrophic overfitting. The main difference between -scheduling and the proposed method is that while -scheduling uniformly applies the same magnitude of the perturbation for every images, the proposed gradually increases the magnitude of the perturbation appropriately  considering the loss surface of each image. Therefore, contrast to -scheduling, the proposed method successfully prevents catastrophic overfitting, even though the same size of the average perturbation is used in the training process.

Adversarial Robustness
In this section, we conduct a set of experiments on CIFAR10 and Tiny ImageNet, using PreAct ResNet-18 (He et al. 2016). Input normalization and data augmentations including 4-pixel padding, random crop and horizontal flip are used.
Here, to observe if catastrophic overfitting occurs, we set the epoch to 200, which is long enough. We use SGD with a learning rate of 0.01, momentum 0.9, weight decay 5e-4 and the learning rate decays with a factor of 0.2 at 60, 120 and 160 epochs. All experiments are performed on a single NVIDIA TITAN V over three different random seeds.
In the training session, the maximum perturbation is set to 8/255. For PGD adversarial training, we use a step size α = max(2/255, /n) where n is the number of iterations. Similarly, TRADES uses α = 2/255 and seven steps for generating adversarial images. Following (Wong, Rice, and Kolter 2020), we use α = 1.25 for fast adversarial training and the proposed method. A regularization parameter β for gradient alignment for GradAlign is set to 0.2, which is the same as the default value given by Andriushchenko and Flammarion (2020).
We then evaluate the robustness of PGD Adv., TRADES, Fast Adv., GradAlign, and our proposed method on the test data at the end of the training (epoch 200). FGSM and PGD50 with 10 random restarts are used for evaluating the robustness of the models. Moreover, to estimate accurate robustness and detect gradient obfuscation (Athalye, Carlini, and Wagner 2018), we also consider PGD50 adversarial images generated from Wide-ResNet 40-10 (Zagoruyko and Komodakis 2016) trained on clean images (Black-box), and AutoAttack (AA) which is one of the latest strong adversarial attacks proposed by Croce and Hein (2020). Table 1 and 2 summarizes the results. From Table 1, we found that multi-step adversarial training yields more robust models, but generally requires more computational time. Especially, TRADES requires over 15 hours which is five times slower than the proposed method. Among single-step adversarial training methods, Fast Adv. is computationally efficient, however, since catastrophic overfitting has occurred, it showed 0% accuracy against strong attacks (PGD50 and AA). The proposed method, by contrast, showed the best standard accuracy and robustness against PGD50, Black-box, and AA with less computational time. GradAlign also provided sufficient robustness; however, it showed less robustness than the proposed method for longer training epoch than Andriushchenko and Flammarion (2020) used. Above all, GradAlign took three times more than the proposed method. As in Table 2, similar results were found on Tiny ImageNet. We included the results of main competitors, PGD2 Adv., TRADES, Fast Adv., and GradAlign with the proposed method with c = 3 which shows the best performance on CIFAR10. Here, again the proposed method shows high standard accuracy and adversarial robustness close to PGD2. However, fast adversarial training and GradAlign show near 0% PGD and AA accuracy as we confirmed catastrophic overfitting in Figure 5.

Conclusion
In this paper, we study the underlying reason for catastrophic overfitting, and empirically showed that it is closely related to decision boundary distortion by analyzing their loss surface and robustness during training. Decision boundary distortion provides reliable understanding for the phenomenon that the catastrophic overfitted model becomes vulnerable to multi-step attacks, while having high robustness on the single-step attacks. Based on these observations, we suggested a new simple method that determine the appropriate magnitude of the perturbation for each image. Further, we evaluated the robustness of the proposed method against various adversarial attacks and provided sufficient robustness using single-step adversarial training without the occurrence of any catastrophic overfitting.

A Distortion and nonlinearity of the loss function
Here, to detect decision boundary distortion numerically, we propose a new measure γ for the nonlinearity of the loss function. In Section 4, we mentioned that single-step adversarial training leads a high curved loss surface. Thus, single-step attacks like FGSM can not generate strong adversarial examples because the following linearity assumption is not satisfied.
To determine the nonlinearity of the loss surface, we define a nonlinearity measure γ based on Equation 9: In this appendix, unless specified otherwise, the same settings in Section 5 are used. Figure  In this section, we aim to provide more figures to describe catastrophic overfitting. First, we plot various shapes of distorted decision boundary. We note that it is not difficult to find distorted decision boundary of the catastrophic overfitted model. Figure 9 shows distorted decision boundary of the fast adversarial trained model on the first four images in the training set of CIFAR10. Although we applied random crop and padding, the loss surface has a distorted interval in the direction of their FGSM adversarial perturbation. In addition, we plot distorted decision boundary in the direction of perturbation obtained by the attack in fast adversarial training in Figure 10. Similar results were found on the test set of CIFAR10. Figure 11 shows the loss surface in FGSM adversarial direction and Figure 12 corresponds to the loss surface in the direction of perturbation obtained by the attack in fast adversarial training.

B.2 Tiny ImageNet
At last, we plot distorted decision boundary of fast adversarial trained model on Tiny ImageNet. Figure 13 and Figure  14 show the results on the training sets and Figure 13 and Figure 14 on the test sets.         Although we mainly focused on fast adversarial training, we also found similar results on FGSM adversarial training. The left plot (a) of Figure 17 shows the distortion and the robust accuracy against FGSM and PGD7 on the training sets. As shown in Figure 17, catastrophic overfitting occurs more frequently on FGSM adversarial training than fast adversarial training. Considering FGSM outputs same adversarial images for the same inputs, the loss function is more easier to be distorted because it is more hard to train images in the distorted interval. The right plot (b) of Figure 17 illustrates the existence of three different labels in the direction of FGSM perturbation in total. In the bottom plot, the distortion (green) is measured on the whole training images. Plot (b) shows distorted decision boundary after catastrophic overfitting. v 1 is a direction of the FGSM adversarial perturbation and v 2 is a random direction.

C.2 Varying the number of checkpoints and the maximum perturbation
Here, we add the results of varing the number of checkpoints c. Figure 18 demonstrates the standard accuracy and the robust accuracy against PGD50 with 10 random restarts. First of all, the proposed method successfully avoids catastrophic overfitting compared to fast adversarial training. Again, we note that fast adversarial training shows 0% accuracy against PGD. The higher c is, the larger standard accuracy is observed. However, we could not find no link between c and the robustness. Table 3 shows the total training time for each number of checkpoints c.   We also tested various maximum perturbation in Figure 19. All methods, including FGSM adversarial, achieve nearly 100% of PGD7 accuracy under = 5/255. However, as increases, FGSM and fast adversarial training could not avoid catastrophic overfitting one after another. The proposed method shows stable training robust accuracy for all values of .   Previously, the piecewise learning rate was used in a large proportion, but since Smith and Topin (2019) discovered the cyclic learning rate helps the model to converge with small epochs, many researchers have adopted the cyclic learning rate schedules. Thus, we perform additional experiments with the cyclic learning rate schedules. The cyclic learning use a maximum learning rate as 0.3 and linearly continue to increase until 15 epochs out of 30 epochs.
First, we note that only a few epochs are normally used to train models using the cyclic learning rate schedule so that there is a small chance of catastrophic overfitting. This implies that the proposed method can not show the strength of preventing catastrophic overfitting. As in Table 4, among single-step adversarial training methods, GradAlign shows the best performance with 46.9% against PGD50. However, it is important to ensure that the average perturbation E[||δ|| ∞ ] of the proposed method did not converged to the maximum perturbation unlike the one using stepwise learning rate scheduling. Indeed, the proposed method achieved 40.2% (+1.1%p) and 39.5% (+1.7%p) for 60 training epochs.